Protobuf message format for Kafka - Connect_CDC - connect_cdc_mimix_share - 6.x

Protobuf (Protocol Buffers) is a message format option for Kafka targets in Connect CDC. You select it from the Message Format drop-down in the Kafka Server Properties dialog alongside the existing CSV, JSON, and Schema Registry (Avro) options.

Connect CDC registers Protobuf schemas automatically in Confluent Schema Registry and serializes Change Data Capture (CDC) events as compact binary Protobuf messages. The schema is stored separately from the message data.

Supported sources

The following source databases support replication to Kafka with Schema Registry (Protobuf):

• IBM i
• Db2/z
• Oracle
• SQL Server
• PostgreSQL
• Db2 LUW

Supported configuration options

When using Schema Registry (Protobuf) as the message format, you can configure the following options in the Server Properties dialog:

• Partition scheme: Source Table Name, Distribution Key, or Round Robin.
• Images for update: After image only, Both in the same record, or Each in a separate record.
• Tombstone handling: Connect CDC publishes DELETE operations as tombstone records (null value) when Partition is set to Distribution Key.
• Schema evolution: When you add a column to a source table and update the model in Director, Connect CDC registers a new schema version for the existing Protobuf topic in the Schema Registry. Field numbers are preserved across schema versions to ensure backward compatibility. New columns are assigned sequential field numbers after the existing fields. Removing a column from the source does not remove it from the schema; it remains as an optional field set to its default value.
• Schema Registry integration: Connect CDC registers Protobuf schemas automatically in Confluent Schema Registry.

Data type mapping

Connect CDC maps source column types to Protobuf field types when generating a schema. The following rules apply to all supported sources:

• Nullable columns use Google Wrapper types (google.protobuf.Int32Value, google.protobuf.StringValue, and so on) declared with the optional keyword, enabling consumers to distinguish between a field explicitly set to its default value and a field not set (SQL NULL). Non-nullable columns use plain proto3 primitive types. This differs from Avro, which uses union types for nullability.
• Decimal and numeric types map to string to preserve precision.
• Date values map to google.type.Date; timestamp values map to google.protobuf.Timestamp.
• Integer types map to the native Protobuf integer type that matches the source precision (int32 or int64).

For complete per-source mapping tables, see Supported datatypes in the System Reference Guide.