Apply and replicator engine overview - connect_cdc_sqdata - Latest

Connect CDC (SQData) Change Data Capture

Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect CDC (SQData)
Version
Latest
Language
English
Product name
Connect CDC (SQData)
Title
Connect CDC (SQData) Change Data Capture
Copyright
2024
First publish date
2000
Last edition
2024-09-05
Last publish date
2024-09-05T15:00:09.754973

The Apply and Replicator Engines can process data from Connect CDC (SQData) capture agents running on any supported platform. The communication between Capture/Publisher and Engine is managed to ensure that only complete Units of work are committed to Targets ensuring data integrity between source and target.

The Apply Engine can also be fed by IBM MQ messages and direct TCP/IP communication with other Apply Engines. They can also read and write, or as we say "Apply" to any datastore supported by the Capture agents:
  • IMS
  • Relational Databases (Db2, Db2/LUW and Oracle)
  • VSAM Files
  • Sequential File on any platform

The Apply Engine can also read and write to most other databases using ODBC including SQL Server, MySQL and PostgreSQL including cloud platforms such as AWS RDS PostgreSQL.

The Apply Engine scripting language can perform virtually any data filtering and transformation imaginable including bit level manipulation. It automatically performs source to target code page translation and can problematically produce reports describing its processing.

The Replicator Engine is designed to operate like a utility, either replicating identical Source/Target datastores (LIMITEDto Db2/z and Db2/LUW today) or replicating IMS and Db2/z data to streaming targets like Kafka and HDFS without any "record level" instructions or manual scripting. In the case of Db2/z the Replicator automatically adapts to source table alterations, modifying JSON schemas automatically or if AVRO is specified, maintaining the Confluent Schema Registry.