As previously stated, Precisely does not recommend the use of Source exception datastores in a production environment. The potential volume of invalid data in legacy source datastores can generate very large amounts of exception data. Recycling the exception data will generally require a different set of control procedures (Jobs, JCL, shell scripts, etc) that are rarely used and themselves subject to becoming obsolete over time.
- The modifications required for the script to handle the invalid data condition also make it possible to restart processing with the record that caused the exception and ensures that all future cases of the invalid data condition will be handled without further exceptions and without using an exception datastore.
- Reprocessing the exception data, once the Apply Engine script has been modified to handle the invalid data condition can be more disruptive to the operational environment than simply modifying the script and restarting the original Apply Engine.
- If it is determined that the data in the source datastore must be corrected, it may still be necessary to process the invalid data as the best or only way to propagate all the changes that may have resulted from the larger change event that uncovered the invalid data in the first place.
Whether you are recycling exception data in the test environment as recommended or if it was determined that the recycling of production exception data is the best way to recover from an invalid data condition, make sure you consider these factors when constructing the process to consider in the construction of the process:
Capturing data exceptions and reprocessing exception data is accomplished the same way whether source data is from an extract process, near real-time capture or asynchronous capture. The script must first be modified to handle the invalid data condition and then the SQDCOPYQ utility (see the MQ Utilities Guide for details on how to use the utility) is used to recycle the records in the exception datastore back through the Apply Engine.
One additional consideration for data captured from IMS or Relational source datastores is the Unit of Work (UOW). In addition to recycling the exception data, you must also recycle the DB2 commit and rollback information into the Unit of Work queue. The UOW queue contains commit and rollback information for each unit-of-work and this information must be copied from the original CDC queue which contains all the actual changed DB2 data records including those that were valid. For detailed information on how the DB2 capture works see the DB2 Capture Reference. Below is an example of how to recycle DB2 exception records.
//*---------------------------------------------------------------------
//** RECYCLLE THE EXCEPTION QUEUE INTO THE SQDATA QUEUE
//*---------------------------------------------------------------------
//SQDCOPYC EXEC PGM=SQDCOPYQ,
// PARM='MQS://MQP/SQDATA.FAILURE.QL
// MQS://MQP/SQDATA.DB2.QL'
//*
//SYSPRINT DD SYSOUT=*
//*---------------------------------------------------------------------
//** COPY DATA FROM A DB2 CDC QUEUE INTO THE UOW QUEUE
//** USING THE "-U" OPTION
//*---------------------------------------------------------------------
//SQDCOPYI EXEC PGM=SQDCOPYQ,
// PARM='MQS://MQP/SQDATA.DB2.QL
// MQS://MQP/SQDATA.DB2.UOW -U'
//SYSPRINT DD SYSOUT=*
//*