Apache's Distributed Streaming Platform, Kafka, is a robust tool for building real-time streaming data pipelines on Open Systems platforms. Its functionality is similar to IBM's MQ and more robust than Java Messaging Service (JMS) however it is not natively supported on z/OS. Connect CDC (SQData) bridges the gap between z/OS and Kafka with its DB2 and IMS Change Data Capture on Z/OS, Replicator Engine running on Linux and Apply Engine Kafka Consumers running on Linux. This provides a hybrid solution that effectively supports three of Kafka's four core API's without modification of z/OS business applications in heterogeneous z/OS and Open Systems environments:
- The Producer API which allows a business application to publish a stream of events to one or more Kafka topics.
- The Connector API that can connect Kafka topics to existing source databases. For example, a connector to a relational database that captures every change to a table.
- The Consumer API that can connect the Apply Engine to a partitioned stream of CDC Data to facilitate Parallel Apply processing to Relational databases and other Targets.
Connect CDC (SQData) high performance Change Data Capture can be used to detect business events that "Change Data" in real time and/or capture all changes that occur in a DB2 source database. The resulting CDC data is processed by the Replicator Engine running on Linux that automatically generates JSON Schemas, translates all source data to UTF-8 and writes that to Kafka Topics using the Producer API.
Connect CDC (SQData) also seamlessly supports Confluent's proprietary Schema Registry by automating the generation of AVRO Schema's for new Tables and updating those schemas when they change in the Relational source database.
