Automatic Protobuf schema registration - connect_cdc_sqdata - Latest

Connect CDC (SQData) generates and registers Protobuf schemas automatically. No manual schema authoring is required. Schemas are derived directly from source record definitions — Db2 DDL for Apply Engine scripts, or the source table catalog metadata provided by the Capture agent for the Replicator Engine.

Because a Confluent Schema Registry is always required for Protobuf targets, automatic schema registration is always active. There is no schema-registry-free Protobuf option.

Automatic registration is a two-step process. When the Apply Engine is parsed — either using the two-step parse-and-run method or the combined SQDENG method — the engine first retrieves each schema from the Confluent Registry that matches the descriptions in the engine script. It then compares those schemas to the descriptions using the TOPIC and SUBJECT values specified for each description. If they are compatible, the parse phase completes and the engine is ready for execution. Any retrieved schema that does not match the current description is automatically registered, the parse phase completes, and the engine is ready for execution.

Example — Db2 to Kafka Protobuf

----------------------------------------------------------------------
-- Name: DB2TOKAF:  Z/OS DB2 To Kafka Protobuf on Linux
----------------------------------------------------------------------
-- Change Log:
----------------------------------------------------------------------
-- 2019-02-01 INITIAL RELEASE using JSON
-- <date>     Switch to Protobuf with Confluent Schema Registry
----------------------------------------------------------------------
OPTIONS
   CDCOP('I','U','D')
  ,CONFLUENT REPOSITORY 'http://schema_registry.precisely.com:8081'
;
----------------------------------------------------------------------
BEGIN GROUP DB2_SOURCE;
DESCRIPTION DB2SQL ./DB2DDL/EMP.ddl AS EMPLOYEE
            KEY IS EMP_NO
            TOPIC IVP_HR_EMPLOYEE
            SUBJECT IVP_HR_EMPLOYEE-value;
DESCRIPTION DB2SQL ./DB2DDL/DEPT.ddl AS DEPARTMENT
            KEY IS DEPT_NO
            TOPIC IVP_HR_DEPARTMENT
            SUBJECT IVP_HR_DEPARTMENT-value;
END GROUP;
----------------------------------------------------------------------
DATASTORE kafka:///*/key
  OF PROTOBUF FORMAT CONFLUENT
  AS TARGET
  DESCRIBED BY GROUP DB2_SOURCE
;
REPLICATE (TARGET)

Protobuf field names automatically preserve their original case from the source definition. Unlike AVRO, no naming transformation option is applied.

Schema evolution for the Replicator Engine works identically to AVRO: when a source table structure changes, the engine automatically regenerates the Protobuf schema and updates the Confluent Schema Registry, with no manual intervention required.