The Quickstart approach is a step by step guide to the installation, configuration, testing and operation of the Connect CDC (SQData) Capture and both the Apply and Replicator Engine components that will create Kafka Topics:
- Determine your initial Kafka Topic data requirements
- Preparation of the Source Data Capture and Target Apply Engine environments
- Configure Engine Controller Daemon
- Determine how you will control the structure of the target Kafka Topics
- Create either Kafka Topic no-map replication Apply Engine script or a Replicator Engine Configuration script
Once these steps have been completed you will then be able to run an end to end test of each of the components in standalone mode. This allows you to work out any security or environmental issues before running alongside other Engines in a shared Capture/Publisher configuration.
After all components are working properly and your first Kafka Topic has been populated successfully, you are ready to add more source/target interfaces and Kafka topics to your configuration.
This Quick Start is intended to supplement, not replace, other documents including the various Data Capture and the Apply and Replicator Engine Reference documentation. We recommend you familiarize yourself with the Precisely MySupport portal where you can learn more about Connect CDC (SQData)'s overall Architecture and approach to Change Data Capture. The answer to many of the questions that inevitably arise during initial installation, configuration and testing will be found in those documents.
Processing data placed into Kafka is beyond the scope of this Quick Start as are the myriad of tools that can be used to "consume" that data. In one example a customer wanted to use Elastic Search for analytics using "LogStash" to process the source data. Their source data was however mainframe application databases that don't naturally produce a usable source of data. Precisely recommended Kafka as an intermediate datastore because of its performance, fault tolerance and because it places data into the open systems platform environment where Elastic Search operates.
There are many tools available to facilitate the Kafka / ES interface, some are proprietary and others are open source, including but certainly not limited to: