Successful Configuration Script creation is similar to any type of application development effort. You must first determine what you are trying to accomplish, design your approach, develop the solution and then perform the required testing activities. Fortunately, Replicator script development requires much less effort than even SQData Apply Engine scripts; In part because the source data schema's are already defined and the Replicator Engine will automatically generate the target schema data structures. The biggest challenge is usually implementing the target datastore infrastructure if you are in the early stages of implementing Kafka or Hadoop with or without Confluent.
It is recommended that you follow the steps in the order presented below.
-
Identify Source Platforms and Relational Tables
Both the host machine, database instance and table names will be required. Since the SQData capture will also have to be implemented, much of this work may have already been accomplished. See the applicable Capture Reference for the steps required to install and configure Change data capture and communications components.
-
Identify Target Datastores
Determine the target datastore, be it Kafka, HDFS or simply Flat Files. If you will be writing Kafka topics, reach out to the internal organization that supports your kafka cluster infrastructure as they will likely have developed standards including any necessary data encryption requirements and templates for "Producers" which is the function of the Replicator Engine.
-
Determine Special Processing Requirements
Determine if any sort of Initial load processing is required. If it is consult with Precisely on the alternative methods for performing that task.
- Create the Replicator Configuration Script
- Validate the Configuration Script