The nature of the IMS source database and the intended use of the Unload utility must be considered when determining the schedule of the SQDIMSU utility. The Utility can be run as either a BMP or Batch process. The examples below illustrates these considerations:
Example 1 - IMS to RDBMS - second pipeline
Initial load or refresh from one or more IMS Databases to an RDBMS using the recommended new LogStream and cloned versions of the CDCzLog Publisher and Apply Engine.
The time required by the initial load will impact the latency of any targets already in an operational replication because CDC Publishing must be stopped during the unload/load process. Ideally the CDC publisher will report no latency at the start of the process, meaning that the existing source database changes have all been replicated.
- Copy any previously created Initial Load/Refresh Engine so that it can be reused or used a source of parts for later versions - Backup.
- Create new Initial Load/Refresh Engine from a copy of current CDC Apply Engine to ensure that any modifications made to CDC Apply Engine are present in new version. Tweak Engine script as required including any special target initialization requirements - New/Modified Engine.
- Configure the new Publisher, add to zOS SQDAEMON configuration - One time activity.
- Configure the local Engine SQDAEMON used to Start/Stop the new Initial Load/Refresh Engine - One time activity.
- Confirm the DBD is configured for Capture (this is critical when a new DBD will be captured for the first time).
- Since the target is an RDBMS and it is a Refresh and the CDC Apply Engine target DATASTORE is configured with the FOR CHANGE then you must drop and recreate the existing target table(s). That will allow compensation to fully resynchronize the Source and Target. - Each time.
- Stop the IMS Log Capture or the CDCzLog Publisher (for the IMS TM or Log Capture), this will cause the CDC Apply Engine to terminate.
The IMS database remains online - No change.
The application continues to performs its normal DL/I BMP or Batch processing - No change. - Start the Unload Publisher and the Initial Load/Refresh Engine -New Job / Engine.
- Start the Unload Utility for the IMS DBD(s) to be unloaded. One or more DBD's can be unloaded at the same time, using multiple Unload Utility Jobs. The SQDIMSU utility is run as an DL/I Batch job - New Jobs, dependent on the successful completion of application batch processing.
- Confirm that all unloaded data has been published and processed by the Initial Load/Refresh Engine.
- Restart the IMS Log Capture or the CDCzLog Publisher (for the IMS TM or Log Capture) and CDC Apply Engine.
- Monitor for compensation, if applicable by periodically reviewing the the Apply Engine report.
Example 2 - IMS to Kafka (messaging targets) - second pipeline
Initial load or refresh from one or more IMS Databases to Kafka (messaging targets) using the recommended new LogStream and cloned versions of the CDCzLog Publisher and Apply Engine.
The time required by the initial load will impact the latency of any targets already in an operational replication because CDC Publishing must be stopped during the unload/load process. Ideally the CDC publisher will report no latency at the start of the process, meaning that the existing source database changes have all been replicated.
Initial load or refresh from one or more IMS Databases using the recommended new LogStream and cloned versions of the CDCzLog Publisher and Apply Engine.
The time required by the initial load will impact the latency of any targets already in an operational replication because Publishing must be stopped during the unload/load process. Ideally the CDC publisher will report no latency at the start of the process, meaning that the existing source database changes have all been replicated.
- Copy any previously created Initial Load/Refresh Engine so that it can be reused or used a source of parts for later versions - Backup.
- Create new Initial Load/Refresh Engine from a copy of current CDC Apply Engine to ensure that any modifications made to CDC Apply Engine are present in new version. Tweak Engine script as required including any special target initialization requirements including code value used to distinguish Load/Refresh from CDC_OP codes (IUD) - New/Modified Engine.
- Configure the new Publisher, add to zOS SQDAEMON configuration - One time activity.
- Configure the local Engine SQDAEMON used to Start/Stop the new Initial Load/Refresh Engine - One time activity.
- Confirm the DBD is configured for Capture (this is critical when a new DBD will be captured for the first time).
The IMS database remains online - No change.
The application continues to performs its normal DL/I BMP or Batch processing - No change.
- Start the Unload Publisher and the Initial Load/Refresh Engine -New Job / Engine.
- Start the Unload Utility for the IMS DBD(s) to be unloaded. One or more DBD's can be unloaded at the same time, using multiple Unload Utility Jobs. The SQDIMSU utility is run as an DL/I Batch job - New Jobs, dependent on the successful completion of application batch processing.
- Confirm that all unloaded data has been published and processed by the Initial Load/Refresh Engine.
- Confirm downstream Kafka Consumer has properly processed the Initial Load/Refresh messages.
Example 3 - IMS to RDBMS - existing pipeline
Initial load or refresh of one or more IMS Databases using the existing logstream, Publisher and Apply Engine.
The time required by the initial load will impact the latency of any databases already in an operational replication because either the IMS Log Capture orALL IMS Application database activity (when the IMS TM Exit Capture is in use), must be stopped during the unload/load process. Ideally the publisher will report no latency at the start of the process, meaning that the existing source database changes have all been replicated.
- Confirm the DBD is configured for Capture (this is critical when a new DBD will be captured for the first time).
- If the target is an RDBMS and it is a Refresh and the CDC Apply Engine target DATASTORE is configured with the FOR CHANGE then you must drop and recreate the existing target table(s). That will allow compensation to fully resynchronize the Source and Target.
- Stop the IMS Log Capture or if using the IMS TM Exit Capture, take the IMS applications or databases offline.
The application performs its normal DL/I Batch processing, if applicable - No change.
- Confirm that the CDCzLog Publisher is still running and the CDC Apply Engine is still connected.
- Start the Unload Utility for the IMS DBD(s) to be unloaded. One or more DBD's can be unloaded at the same time, using multiple Unload Utility Jobs. The SQDIMSU utility is run as an DL/I Batch job - New Jobs, dependent on the successful completion of application batch processing.
- Confirm that all unloaded data has been published and processed by the CDC Apply Engine.
- Restart the IMS Log Capture or if using the IMS TM Exit Capture, bring the IMS applications or databases back online.
- Monitor for compensation, if applicable by periodically reviewing the Apply Engine report.Note: In this example, the sequence of database updates and Unit-of-Work integrity are maintained because the CDC records created by the utility will all be published to the Apply Engine before the "first" or "next" CDC record from application database is replicated. While there may be some latency as the utility's "Insert" are processed, the Apply Engine will rapidly catch up, performing compensation as necessary and resuming near real-time latency for subsequent on-line updates.