A configuration file is used to pass properties to Kafka and Schema Registry connections made from Connect CDC.
The properties in the configuration file are standard to, and interpreted by, the Kafka or Schema Registry client libraries. Therefore, the configuration file passed via Connect CDC only requires properties that would also be required from any Java program using the same Kafka/Schema Registry client jar files. The only property that is specific to Connect CDC that may be required in the configuration file is the connect.schema.registry property. This property is needed by Connect CDC to determine which flavor of schema registry is being used.
The required properties and their expected values can vary depending on:
- The customizations specified while configuring the Kafka brokers or schema registry server.
- Kafka or schema registry distribution being used such as Apache, Confluent on-prem, Confluent Cloud, Hortonworks, Amazon MSK, etc.
Refer to the Kafka vendor’s documentation for more information about available properties and their expected values.
When to use the configuration file
A configuration file for Kafka is only required in the following scenarios:
- There are non-default configuration properties required by the Kafka brokers for establishing a connection. For example, when brokers are configured for SSL, Kerberos, or username/password authentication.
- Schema registry is being used
- Customizations are required to allow Kafka producers to create or write to topics
- Performance-tuning properties are desired