Create a new server for Kafka - Connect_CDC - aws_mainframe_modernization_service - connect_cdc_mimix_share - 6.x

Connect CDC Getting Started Guide

Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect CDC (MIMIX Share)
Version
6.x
Language
English
Product name
Connect CDC
Title
Connect CDC Getting Started Guide
Copyright
2024
First publish date
2003
Last updated
2024-10-15
Published on
2024-10-15T20:38:41.117981
This section describes how you create a new server for Kafka.
  1. Select the EDMM and select New Server from the context menu.

  2. On the Servers Properties dialog select Kafka from the dropdown list in the DBMS type field.

  3. Enter information for each field on the Server Properties dialog. Refer to Server Properties dialog for Kafka for field information.

     

Table 1. Server Properties dialog for Kafka

Parameter

Description

Message Format

Sets the format for messages passed to Kafka:
  • CSV (No Schema): Comma Separated Values
  • Json: encodes messages in json objects, rather than json frag­ments, readable by a json parser
  • Schema Registry (Avro): encodes messages in Apache Avro for­mat, which conforms to field names and types defined in the schema registry

Omit null columns

Discards data resulting in a null value from the message. Only available for targets set to use the Json message format.

Multiple Tables to 1 Topic

If selected, Connect CDC maps all tables to the specified name in the topic field. Available when messages are sent in CSV or Json format.

Topic

A Kafka topic name to which Connect CDC writes changes.

Key

Key determines the Topic partition behavior:
  • Source Table Name: Partitions are divided up by table name.
  • Round Robin: default partitioning behavior that partitions the record on regular size intervals

Prefix Table Name

If selected, prepends the source table name as a prefix to the Kafka message.

1 Topic Per Table

If selected, Connect CDC only maps one table to any single topic. Available when messages are sent in Json or Schema Registry (Avro) format.

Brokers

Comma separated list of names of Kafka nodes with active ports in the Kafka cluster.

For example: staging1-master.hostfarm.net:6667

Config File

Path to the local Kafka configuration file that contains Schema Registry URL.
Connect CDC typically detects the Schema Registry vendor based on the installed client jar libraries. To set a specific vendor for schema registry, add the following parameters to the Kafka configuration file. For more information see, Sample kafka configuration.
  • For Confluent: connect.schema.registry=CONFLUENT

  • For Hortonworks: connect.schema.registry=HORTONWORKS

The Kafka configuration file does not contain Kerberos information.

Use Kerberos

Enables Kerberos authentication support. Kerberos settings include:
  • Principal: user name for Kerberos authentication

  • Keytab: path to the Kerberos keytab file

More options ... tab  

 

Images for update

Sets how Connect CDC writes Before and After values for all columns to the Kafka Topic when it captures an update statement:
  • After image only: Default. Connect CDC only writes After values to the record.

  • Both in the same record: Connect CDC writes both Before and After values to the same record. Only available for targets set to use the Json message format.

  • Each in a separate record: Connect CDC writes both Before and After values to different records. Connect CDC sets the before value entry in the sv_manip_type column to B.

Always send LOB

Always writes a Large Object (LOB) after image row whether or not an update causes a change. If the LOB column value contains a change, the value in the before image row contains an empty string.

  • No change value: Sets the value written to the LOB column value in the before image row when the LOB column value has not changed. This value only applies when you select Always send LOB. For After image only option, the no change value is not used. Connect CDC ignores this setting when Images for update is set to After image only.

Max Lob size

Sets the maximum size in bytes of Large object (LOB) values written to Kafka, which has a default message size of 1MB. The default value of this parameter is 50000 bytes (5 KB), and valid values range from 100 bytes to 2GB.

If a LOB column value is larger than the Max Lob size value, Connect CDC truncates the value to the maximum LOB size before committing it Kafka and issues a warning message to the kernel log.

Asynchronous Options

Enables Connect CDC to apply row changes to a Kafka target asynchronously, which means Connect CDC submits changes to Kafka without waiting for the execution results for each row. This also means Connect CDC does not calculate the execution results for each change until either (a) the number of rows changed reaches the number of rows set by the Cache Size Limit, or (b) there are no more rows to apply. This approach has been shown to significantly increase performance, but Connect CDC does not apply rows in the batch resulting in an error and reports the error when it checks execution results, not when it submits the change to Kafka.

Note: Do not use asynchronous apply for data connections that require immediate error handling or reporting; Connect CDC does not log errors or resubmit data if a row change fails due to errors other than a lost connection.

Cache Size Limit

Sets a limit on the number of rows Connect CDC buffers before calculating the execution results when applying changes asynchronously. The default value is 1000.

Use Azure Keyvault Enable this option to connect to external servers like Kafka, Event hub, and Source data systems. Communication between Connect and Azure Key Vault is secured by using TLS.
  • Keyvault Name: A string value that uniquely identifies a key vault in the service.
  • Secret Name: A unique identifier to store and protect secrets.