Kcat utility - connect_cdc_sqdata

The open source Kcat utility is a generic command line non-JVM Apache Kafka producer and consumer utility. While it is best known for operating as both a Producer and a Consumer, it provides another important function when starting to test a Connect CDC (SQData) Apply or Replicator Engine; Kcat is written in C rather than Java and therefore exercises the same libraries and configuration prepared for the Engine.

You can use Kcat to inspect and create data in Kafka. You can find Kcat on github at https://github.com/edenhill/kcat. You may find others are already using it inside your organization or that it is already installed on your system.

One of the many functions of Kcat is to quickly confirm your configuration without having to connect to a publisher for test data:

In Producer mode, Kcat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).
In Consumer mode, Kcat reads messages from a topic and partition and prints them to stdout using the configured message delimiter.
Kcat features a Metadata list (-L) mode to display the current state of the Kafka cluster and its topics and partitions.
Most importantly, it can also be configured to use the sqdata_kafka_producer.config file.

Finally, it also supports Avro message deserialization using the Confluent Schema-Registry when you are ready to move up to AVRO from JSON, and generic primitive deserializers.

Example 1

Connect as a Consumer to "my_broker", read the last ten (10) messages from the topic named "my_sqdata_test_topic" and then exit.

kcat -C -b my_broker -t my_sqdata_test_topic -p 0 -o -10 -e

Example 2

Connect as a Consumer to the default broker, read and display all the messages from the topic "my_sqdata_test_topic" with the output in hex.

kcat -C -b 0 -t my_sqdata_test_topic -o 0 -e | hexdump

Example 3

Connect as a Producer using the same configuration used by the Apply or Replicator Engine as specified in the sqdata_kafka_producer.config file. Continue to run until interrupted by CTL-C.

kcat -P -F ./<path_to>/sqdata_kafka_producer.config  -t my_sqdata_test_topic  -o 0

Example 4

Decode Avro key (-s key=avro), value (-s value=avro) or both (-s avro) to JSON using schema from the Schema-Registry.

kcat -b my_broker -t my_sqdata_test_topic -s avro -r http://schema-registry-url:8081

Kcat utility - connect_cdc_sqdata - Latest

Connect CDC (SQData) Kafka Quickstart