Kafka AVRO confluent target - connect_cdc_sqdata - Latest

Connect CDC (SQData) Replicator Engine

Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect CDC (SQData)
Version
Latest
Language
English
Product name
Connect CDC (SQData)
Title
Connect CDC (SQData) Replicator Engine
Copyright
2024
First publish date
2000
ft:lastEdition
2024-08-01
ft:lastPublication
2024-08-01T16:58:30.842568

Replication to Kafka in AVRO format using the Confluent convention and the Confluent Schema Registry is the most efficient and highest performance Kafka option. The Confluent AVRO convention prefixes each AVRO message with a small header providing the schema id used to encode the AVRO formatted message. Automatic maintenance of the Schema's in the Registry provides for near automatic schema evolution.

Prerequisites
  • The url of the Confluent Repository to use must be present in the OPTIONS statement.
  • Connect CDC (SQData) assumes that support for command line url (cURL) and the Confluent Schema Registry has been installed and configured somewhere in your environment for use by both Apply and Replicator Engines. The URL and port number of the Registry will be required in order to read and register schema information.
  • The external library libcurl is required to support communication with the Confluent Schema Registry. Libcurl can be installed using distribution provided package if available or built from source which can be downloaded from https://curl.se/download.html.
  • The Confluent Platform is available at https://www.confluent.io/download/.
  • The MAPPINGS section, either static or dynamic, must be able to provide a 'subject', which is effectively a key to the Confluent Schema Registry, for each source object to be replicated. The Replicator will use the description of the source object provided by the Publisher to generate an appropriate schema, and will use the subject to determine the confluent topic id associated with the object, using the provided 'subject' name. If the schema is not registered, the Replicator will register a new one under the indicated 'subject'.
  • The MAPPINGS section may also provide a ‘topic’ which will be used to name the Kafka topic for a specific source object name (i.e. table name). A given topic can be used for one or more object tables, even all of them. If only one topic is to be used, it can be encoded in the target url. If more than one topic is to be used, the target url must be a generic url, of nature Kafka, the substitution parameter "*" being replaced by the topic value for each Source object.

Example

Simple AVRO Confluent formatted Kafka where every source object name (i.e. table name) is written to a specified Topic with a common prefix and suffix and using an AVRO schema managed by the Confluent Registry.
REPLICATE
  DB2 cdc://<host_name>:<sqdaemon_port>/<publisher_name>/<subscription_name>
  TO AVRO CONFLUENT kafka:///<prefix>_*_<suffix>/key
;
OPTIONS
CONFLUENT REPOSITORY 'http://<host_name>:<host_port>',
;
Note:
  • The relationship between the DESCRIPTION Alias, Topic and Subject are matters determined by the planners and architects of the organization's Confluent Schema Registry. The choice of SUBJECT may be made based on the default supported by the Confluent Control Center which requires the SUBJECT to be the same as the TOPIC with the addition of "-value".
  • The Confluent Schema Registry supports multiple Topic naming strategy and all are supported but they may or may not be compatible with other tools including Confluent's own Control Center.