Historically, working with the Apply Engine was always a two step process involving a parse step and an execution or run step. Parsing was generally something that occurred while an engine script was under development and when migrating from one environment to another, for example from Test to Production. The idea was and still is to eliminate the need for changes to an engine script during migration while providing the means to modify critical elements using a "parameter" when re-parsing during migration.
Occasionally, there was a need to provide a parameter at run-time and while that could be accommodated by a two step parse/run shell script or zOS JCL, it has now been simplified using the SQDENG program. Today, SQDPARSE and SQDATA are merely alias names for SQDENG with slightly different control parameters. You can chose which method you prefer.
Syntax
sqdata <engine>.prc [--identity=<path_to/id_nacl>] [--log-level=n] > <engine>.rpt 2>&1| Keyword | Description |
|---|---|
| <engine>.prc | The name of the parsed script file created during the parse process. This file will be used as input to the Apply Engine Engine. The suggested file extension for the parsed script file is in the Multiplatforms environment is prc. |
| [--identity=<path_to/id_nacl>] (Non-z/OS only) | Local file system path and file name or AKV url for the NaCl private key to be used by the Engine. On z/OS platforms both Public and Private Key files are specified at runtime by DD statements. |
| [--log-level=n] (Non-z/OS only) | Diagnostic level of logging, 0=none (not recommended), 2=normal (DEFAULT), 4=warnings, 8=detailed (use only as recommended by Precisely Support and usually in conjunction with "diagnostic" version of product (../bin/dbg). A diagnostic log file named <executible>_<pid>.log will be generated for any log level greater than zero (0). Diagnostic log files may range in size from 2-3 lines to an unlimited number of lines so file system size may need to be monitored. |
| [--max-uows=n] (Linux Kafka targets only) | Optional runtime parameter that overrides the default of 100 in-flight units-of-work (UOWs). Some Kafka (and compatible) targets may be unable to manage the default number of UOWs at the same time. That may be due to capacity issues as well as cloud service subscription level. Consider this an additional tuning parameter besides those specified in the Producer Configuration file. Precisely recommends discussing anticipated workload with the team/provider of the target environment and reviewing the recommended producer settings and if applicable, their relationship to subscription service level. Confirm that recommended producer parm value documentation applies to your current subscription levels. |
| <engine>.rpt | The name of the report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment and to SYSOUT in the z/OS environment. The suggested file extension for this file in the Multiplatforms environment is rpt. |
| [<engine>.rpt] | The name of the Engine Runtime Report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment. The suggested file extension for this file in the Multiplatform environment is rpt. |
| --enable-trxmeta | Turn on writing of transaction meta data fields to a text file provided in OPTIONS Statement for Apply Engine. Note: This command line options will be effective only when TRXMETA file is provided in OPTIONS Statement for Apply Engine.
This can be disabled using command sqdmon tune --disable-trxmeta Warning: There is a potential for very high volumes of output in the TRXMETA file in JSON format.
|
| --enable-latency-report | Turn on latency report generation in SQDMON display command output for Apply Engine, and to the Engine Runtime Report file (<engine>.rpt]) which is created when Apply Engine stops. This can be disabled using command sqdmon tune --disable-latency-report |
Latency reporting
When you run the flag: --enable-latency-report either directly on the command line or using sqdmon tune --enable-latency-report, five (5) additional latency‑related data points are added to the SQData Observability dashboard for Replicator Engine.
- Transactions Acknowledged - Total number of transactions acknowledged at the Apply or Replicator Engine.
- Transaction Capture Latency - Time between when a transaction is written to the source logs and when it is read by Capture.
- Transaction Replication Latency - Time between the Apply/Replicator Engine completing the transaction and the Publisher pushing it to the engine.
- Total Capture Latency - Sum of all Transaction Capture Latency for every transaction processed by the engine.
- Total Replication Latency - Sum of all Transaction Replication Latency for every transaction processed by the engine.
SQDENG offers an alternative to running separate Parse and SQData processes. It reads the <engine>.sqd file, parses it, and if the parse is successful, immediately executes the Engine. This approach is useful when managing many Engines that are identical except for their input parameters. However, Precisely recommends starting with a standalone Parse. This method generates a <engine>.prt file, which contains all the fully expanded components of the engine, shows the substituted parameter values, and provides a visual confirmation of a successful Parse before execution. You can achieve this using SQDENG with the optional parameters --parse and -l <engine>.prt.
Syntax
sqdeng [--script=]<engine>.sqd [<parm1> <parm2> …<parmn>] [--fork=<pnumber_of_ath_to/id_nacl>] [–identity=<path_to/id_nacl>] [-–curl-config-dir=<path_to/sqdata_curl.conf>] [--log-level=n] [-l <engine>.prt] [--max-uows=n] [--parse] [> <engine>.rpt] | Keyword | Description |
|---|---|
| <engine>.sqd | The name of the Apply Engine script file used as input to the parser phase. The recommended file extension for this file in the Multiplatforms environment is sqd. |
| [<parm1> <parm2> …<parmn>] | One or more optional parameters used in the parser phase. See section Substitution Parameters in the Parser Directives section above for an explanation of the two types of parameters supported: Named and Positional. Examples will use the recommend Named parameter syntax. |
| [--fork=<pnumber_of_ath_to/id_nacl>] | Run multiple copies of the engine at once, distributing the input between them intended for high volume loads only. Precisely recommends discussing the use of this parameter with Support before its use. |
| [--identity=<path_to/id_nacl>] (Non-z/OS only) | Local file system path and file name or AKV url for the NaCl private key to be used by the Engine. On z/OS platforms, both Public and Private Key files are specified at runtime by DD statements. |
| --curl-config-dir | Command line option to provide sqdata_curl.conf file path, if not provided will pick from current working directory. The sqdata_curl.conf file has the following options:
Note: sqdata_curl.conf will be picked up from current directory unless path is specified using command line option --curl-config-dir to sqdeng, sqdparse, and sqdrpl.
|
| [--log-level=n] (Non-z/OS only) | Diagnostic level of logging, 0=none (not recommended), 2=normal (DEFAULT), 4=warnings, 8=detailed (use only as recommended by Precisely Support and usually in conjunction with "diagnostic" version of product (../bin/dbg). A diagnostic log file named <executable>_<pid>.log will be generated for any log level greater than zero (0). Diagnostic log files may range in size from 2-3 lines to an unlimited number of lines so file system size may need to be monitored. |
| [-l <engine>.prt] | The name of the optional Parser report file created at the start of execution. If this parameter is not specified, no parser report is produced. The suggested file extension for this file in the Multiplatform environments is .prt. |
| [--max-uows=n] (Linux Kafka targets only) | Optional runtime parameter that overrides the default of 100 in-flight units-of-work (UOWs). Some Kafka (and compatible) targets may be unable to manage the default number of UOWs at the same time. That may be due to capacity issues as well as cloud service subscription level. Consider this an additional tuning parameter besides those specified in the Producer Configuration file. Precisely recommends discussing anticipated workload with the team/provider of the target environment and reviewing the recommended producer settings and if applicable, their relationship to subscription service level. Confirm that recommended producer parm value documentation applies to your current subscription levels. |
| [--parse] | Instructs sqdeng to only parse the script and not attempt execution. Use together with -l to only parse and produce a full Parser report. |
| [> <engine>.rpt] | The name of the runtime report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment. The suggested file extension for this file in the Multiplatform environments is rpt. |
- When using SQDENG the working directory is a bit more critical because references in the script to the location of Source descriptions, called procedures, etc must be sensitive to their location in the directory hierarchy, ie: ./<directory_name>/<file_name>.
- Azure Key Vault (AKV) based secrets in Connect CDC (SQData) are supported only on Linux platforms.
- AKV requires an Azure Active Directory (AAD) token to be presented to retrieve secrets. SQData retrieves AAD tokens from Azure differently when running on on-prem Linux machines and when running on Azure Linux VM.
- When running on-prem, to retrieve AAD, tenant_id, client_id and client_secret have to be specified in the sqdata_cloud.conf file located in the working directory.
- When running on Azure VM, AAD will retrieved from managed identity. Two types of managed identities are supported:
- If client_id is specified in sqdata_cloud.conf file, then AAD token is retrieved from user managed identity
- If tenant_id, client_id and client_secret are not specified, then AAD token is retrieved from system managed identity.
Example 1
Execute an engine script named DB2TOORA.sqd that will perform near real-time replication of "Sales" application data from a DB2 database captured on a z/OS system named "MVS21" to Oracle tables running on a UNIX system. The source "Host" system and DB2 System ID were passed as parameters because the engine script has been written so that it can be reparsed to process data coming from other systems as well.
sqdata ./ENGINE/DB2TOORA.prc 2>&1 | tee ./ENGINE/DB2TOORA.rpt
Execution of the same engine using SQDENG, might look like this:
sqdeng --script=./ENGINE/DB2TOORA.sqd ENGINE=DB2TOORA HOST=MVS21 SSID=DB2T --list=./ENGINE/DB2TOORA.prt > ./ENGINE/DB2TOORA.rpt