Create a new server for Kafka - Connect_CDC - aws_mainframe_modernization_service - connect_cdc_mimix_share

Create a new server for Kafka - Connect_CDC - aws_mainframe_modernization_service - connect_cdc_mimix_share - Latest

AWS Mainframe Modernization - Data Replication for IBM i

Product type

Software

Portfolio

Integrate

Product family

Connect

Product

AWS Mainframe Modernization > AWS Mainframe Modernization Service

Version

Latest

Language

English

Product name

AWS Mainframe Modernization

Title

AWS Mainframe Modernization - Data Replication for IBM i

2024

First publish date

2003

Last updated

2024-02-01

Published on

2024-02-01T23:02:31.099696

This section describes how you create a new server for Kafka.

Select the EDMM and select New Server from the context menu.
On the Servers Properties dialog select Kafka from the dropdown list in the DBMS type field.
Enter information for each field on the Server Properties dialog. Refer to Server Properties dialog for Kafka for field information.

Table 1. Server Properties dialog for Kafka
Parameter	Description
Message Format	Sets the format for messages passed to Kafka: CSV (No Schema): Comma Separated Values Json: encodes messages in json objects, rather than json fragments, readable by a json parser Schema Registry (Avro): encodes messages in Apache Avro format, which conforms to field names and types defined in the schema registry
Omit null columns	Discards data resulting in a null value from the message. Only available for targets set to use the Json message format.
Multiple Tables to 1 Topic	If selected, Connect CDC maps all tables to the specified name in the topic field. Available when messages are sent in CSV or Json format.
Topic	A Kafka topic name to which Connect CDC writes changes.
Key	Key determines the Topic partition behavior: Source Table Name: Partitions are divided up by table name. Round Robin: default partitioning behavior that partitions the record on regular size intervals
Prefix Table Name	If selected, prepends the source table name as a prefix to the Kafka message.
1 Topic Per Table	If selected, Connect CDC only maps one table to any single topic. Available when messages are sent in Json or Schema Registry (Avro) format.
Brokers	Comma separated list of names of Kafka nodes with active ports in the Kafka cluster. For example: staging1-master.hostfarm.net:6667
Config File	Path to the local Kafka configuration file that contains Schema Registry URL. Connect CDC typically detects the Schema Registry vendor based on the installed client jar libraries. To set a specific vendor for schema registry, add the following parameters to the Kafka configuration file. For more information see, Sample kafka configuration. For Confluent: connect.schema.registry=CONFLUENT For Hortonworks: connect.schema.registry=HORTONWORKS The Kafka configuration file does not contain Kerberos information.
Use Kerberos	Enables Kerberos authentication support. Kerberos settings include: Principal: user name for Kerberos authentication Keytab: path to the Kerberos keytab file
More options ... tab
Images for update	Sets how Connect CDC writes Before and After values for all columns to the Kafka Topic when it captures an update statement: After image only: Default. Connect CDC only writes After values to the record. Both in the same record: Connect CDC writes both Before and After values to the same record. Only available for targets set to use the Json message format. Each in a separate record: Connect CDC writes both Before and After values to different records. Connect CDC sets the before value entry in the sv_manip_type column to B.
Always send LOB	Always writes a Large Object (LOB) after image row whether or not an update causes a change. If the LOB column value contains a change, the value in the before image row contains an empty string. No change value: Sets the value written to the LOB column value in the before image row when the LOB column value has not changed. This value only applies when you select Always send LOB. For After image only option, the no change value is not used. Connect CDC ignores this setting when Images for update is set to After image only.
Max Lob size	Sets the maximum size in bytes of Large object (LOB) values written to Kafka, which has a default message size of 1MB. The default value of this parameter is 50000 bytes (5 KB), and valid values range from 100 bytes to 2GB. If a LOB column value is larger than the Max Lob size value, Connect CDC truncates the value to the maximum LOB size before committing it Kafka and issues a warning message to the kernel log.
Enable Asynchronous Apply	Enables Connect CDC to apply row changes to a Kafka target asynchronously, which means Connect CDC submits changes to Kafka without waiting for the execution results for each row. This also means Connect CDC does not calculate the execution results for each change until either (a) the number of rows changed reaches the number of rows set by the Cache Size Limit, or (b) there are no more rows to apply. This approach has been shown to significantly increase performance, but Connect CDC does not apply rows in the batch resulting in an error and reports the error when it checks execution results, not when it submits the change to Kafka. Note: Do not use asynchronous apply for data connections that require immediate error handling or reporting; Connect CDC does not log errors or resubmit data if a row change fails due to errors other than a lost connection.
Cache Size Limit	Sets a limit on the number of rows Connect CDC buffers before calculating the execution results when applying changes asynchronously. The default value is 100.