Run Apply engine - connect_cdc_sqdata - Latest

Connect CDC (SQData) Kafka Quickstart

Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect CDC (SQData)
Version
Latest
Language
English
Product name
Connect CDC (SQData)
Title
Connect CDC (SQData) Kafka Quickstart
Copyright
2024
First publish date
2000
Last edition
2024-07-30
Last publish date
2024-07-30T20:00:09.892433

On the Linux platform, the Apply Engine can be invoked from the command line, inheriting the working directory and other previously established environmental setting, a shell script or using the Engine Controller Daemon.

Syntax

        sqdata <engine>.prc > <engine>.rpt [--parse]
Keyword and Parameter Descriptions
Keyword Description
<engine> Name of the Apply Engine script file used as input to the parser.
[--parse] Optional parameter that invokes the Apply Engine and produces the Schemas for your review.

Example 1

The following command line text invokes the Engine using the Parsed script named DB2TOKAF.prc. The Runtime report is to be written to the file named DB2TOKAF.rpt.
sqdata ./ENGINE/DB2TOKAF.prc > DB2TOKAF.rpt

Example 2

The following shell script invokes the Engine using the same Parsed script file named DB2TOKAF.prc. The Runtime report is again written to the file in the working directory named DB2TOKAF.rpt.
#!/bin/sh
cd ./ENGINE
sqdata DB2TOKAF.prc > ../DB2TOKAF.rpt

Example 3

The Apply Engine can also be started using the sqdmon utility and the previously configured Apply Engine controller daemon. Once again the command can be executed from the command line or using a simple Shell script.
sqdmon START DB2TOKAF
Note:
  • An example of the resulting Kafka Topic for a changed EMPLOYEE table row was illustrated previously in Determine Kafka Output Format.
  • Because the Apply Engine controller daemon was configured to listen on the default TCP/IP port 2626 it is unnecessary to specify the port when running sqdmon. If another port number was used the --service=<port> or -s <port> would be included in the command.
  • The previously mentioned sqdata_kafka_producer.conf file is located in the Working Directory of the Apply Engine at launch therefore, depending on how the engine is started, it may need to be relocated.
  • Apply Engines configured to use AVRO with a Confluent Schema Registry will automatically attempt to retrieve existing registered schemas for all source DESCRIPTIONS at run time. If the schema retrieved matches the one automatically generated by the Apply Engine then processing will proceed. If the schema does not exist or there are differences detected the newly generated schema will be added to the Schema Registry and subsequent Kafka message payloads will be generated based on the new schema. This activity will only occur when this type of Apply Engine is Started. Changes to Source system Schemas will still require coordination to ensure that Engines are Stopped and source Schemas are Updated as part of the normal "Source system Maintenance Schedule".
  • Precisely strongly recommends use of the --parse option before running the engine the first time to ensure the generated Schemas meet your needs and expectations before they are registered in the Confluent Schema Registry. This is the proper was to document and review the schema before they are registered.