Setup Precisely Apply Engine - connect_cdc_sqdata - aws_mainframe_modernization_service - Latest

1. Select a unique apply <engine_name> that should also be used as the jobname in the job script file and the sub directory path in the instance as shown in the following steps for VSMTOKFK.
2. Create the apply engine directories in $SQDATA_VAR_DIR/apply/<ENGINE_NAME>. For example,

   connect> mkdir -p $SQDATA_VAR_DIR/apply/VSMTOKFK/ddl 
   connect> mkdir -p $SQDATA_VAR_DIR/apply/VSMTOKFK/scripts

3. Upload, create, or edit the job script <engine_name>.sqd. For example,

   connect> vi $SQDATA_VAR_DIR/apply/VSMTOKFK/scripts/vsmtokfk.sqd

Example script in the working directory of the Apply Engine

• ./cpy are the working directory where the Copybook of VSAM files are stored
• IP:PORT is the IP address of the mainframe and PORT number on which the daemon process is running on mainframe system
• MSKTutorialTopic is the topic name of Kafka created in earlier step where message will be produced

  -- JOBNAME -- PASS THE SUBSCRIBER NAME
  -- REPORT  progress report will be produced after "n" (number) of Source records processed.
  
  JOBNAME VSMTOKFK;
  OPTIONS
  CDCOP('I', 'U', 'D'),
  PSEUDO NULL = NO,
  USE AVRO COMPATIBLE NAMES,
  APPLICATION ENCODING SCHEME = 819;
  
  --SOURCE DESCRIPTIONS
  BEGIN GROUP VSAM_SRC;
  DESCRIPTION COBOL ../copybk/ACCOUNT AS ACCTDAT;
  END GROUP;
  --TARGET DESCRIPTIONS
  BEGIN GROUP VSAM_TGT;
  DESCRIPTION COBOL ../copybk/ACCOUNT AS ACCTDAT;
  END GROUP;
  --SOURCE DATASTORE (IP & Publisher name)
  DATASTORE cdc://XXX.XX.XX.XX:2626/VSMTOKFK/VSMTOKFK
  OF VSAMCDC
  AS CDCIN
  DESCRIBED BY GROUP VSAM_SRC ACCEPT ALL;
  
  
  --TARGET DATASTORE(s) - Kafka and topic name
  DATASTORE 'kafka:///MSKTutorialTopic/key'
  OF JSON
  AS CDCOUT
  DESCRIBED BY GROUP VSAM_TGT FOR INSERT;
  --MAIN SECTION
  PROCESS INTO
  CDCOUT
  SELECT
  {
    SETURL(CDCOUT, 'kafka:///MSKTutorialTopic/key')
    REMAP(CDCIN, ACCTDAT, GET_RAW_RECORD(CDCIN, AFTER), GET_RAW_RECORD(CDCIN, BEFORE))
    REPLICATE(CDCOUT, ACCTDAT)
  }
  FROM CDCIN; 

• Add stanza to sqdagents.cfg. For example,

  cat $SQDATA_DAEMON_DIR/cfg/sqdagents.cfg

  
  
  
  Note: See sample stanzas in $SQDATA_VAR_DIR/templates/sqdagents.cfg. Consider setting auto_start=1 to have the engine automatically started in case of a system restart.

You can validate the apply engine job script by using the SQData parse command to create the compiled .prc file expected by the sqdata engine.

1. Run this command.

   connect> sqdparse \
   $SQDATA_VAR_DIR/apply/VSMTOKFK/scripts/vsamtokfk.sqd \
   $SQDATA_VAR_DIR/apply/VSMTOKFK/scripts/vsamtokfk.prc

2. Correct any errors identified by the parse command and try again.

Script for Kafka producer configuration file sqdata_kafka_producer.conf in the scripts folder.

cat $SQDATA_VAR_DIR/ apply/VSMTOKFK/scripts/sqdata_kafka_producer.conf 

builtin.features=SASL_SCRAM
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.username=XXXX
sasl.password=XXXX-secret
metadata.broker.list=b-1.msktutorialcluster.XXXXX.XX.kafka.us-east-1.amazonaws.com:9096,b-3.msktutorialcluster.XXXXX.XX.kafka.us-east-1.amazonaws.com:9096,b-2.msktutorialcluster.XXXXX.XX.kafka.us-east-1.amazonaws.com:9096