Historically, working with the Apply Engine was always a two step process involving a parse step and an execution or run step. Parsing was generally something that occurred while an engine script was under development and when migrating from one environment to another, for example from Test to Production. The idea was and still is to eliminate the need for changes to an engine script during migration while providing the means to modify critical elements using a "parameter" when re-parsing during migration.
Occasionally, there was a need to provide a parameter at run-time and while that could be accommodated by a two step parse/run shell script or zOS JCL, it has now been simplified using the SQDENG program. Today, SQDPARSE and SQDATA are merely alias names for SQDENG with slightly different control parameters. Customers can chose which method they prefer.
Syntax
sqdata <engine>.prc [--identity=<path_to/id_nacl>] [--log-level=n] > <engine>.rpt 2>&1| Keyword | Description |
|---|---|
| <engine>.prc | The name of the parsed script file created during the parse process. This file will be used as input to the Apply Engine Engine. The suggested file extension for the parsed script file is in the Multiplatforms environment is prc. |
| [--identity=<path_to/id_nacl>] (Non-z/OS only) | Local file system path and file name or AKV url for the NaCl private key to be used by the Engine. On z/OS platforms both Public and Private Key files are specified at runtime by DD statements. |
| [--log-level=n] (Non-z/OS only) | Diagnostic level of logging, 0=none (not recommended), 2=normal (DEFAULT), 4=warnings, 8=detailed (use only as recommended by Precisely Support and usually in conjunction with "diagnostic" version of product (../bin/dbg). A diagnostic log file named <executible>_<pid>.log will be generated for any log level greater than zero (0). Diagnostic log files may range in size from 2-3 lines to an unlimited number of lines so file system size may need to be monitored. |
| [--max-uows=n] (Linux Kafka targets only) | Optional runtime parameter that overrides the default of 100 in-flight units-of-work (UOWs). Some Kafka (and compatible) targets may be unable to manage the default number of UOWs at the same time. That may be due to capacity issues as well as cloud service subscription level. Consider this an additional tuning parameter besides those specified in the Producer Configuration file. Precisely recommends discussing anticipated workload with the team/provider of the target environment and reviewing the recommended producer settings and if applicable, their relationship to subscription service level. Confirm that recommended producer parm value documentation applies to your current subscription levels. |
| <engine>.rpt | The name of the report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment and to SYSOUT in the z/OS environment. The suggested file extension for this file in the Multiplatforms environment is rpt. |
| [<engine>.rpt] | The name of the Engine Runtime Report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment. The suggested file extension for this file in the Multiplatform environment is rpt. |
SQDENG is an alternative to running independent Parse and SQData processes. It reads the <engine>.sqd file, parses the file and upon a successful parse, immediately executes the Engine. While this works well when a large number of Engines, identical except for the parameters passed are managed, Precisely recommends beginning with a standalone Parse because it produces an <engine>.prt file containing the fully expanded parts making up the engine, displaying substitution parm values and a the visual confirmation of a successful Parse before attempting execution. The can be accomplished using SQDENG with two optional parms --parse and -l <engine>.prt.
Syntax
The following syntax is used for invoking the Apply Engine on linux and AIX in "parse and run" mode.
sqdeng [--script=]<engine>.sqd [<parm1> <parm2> …<parmn>] [--fork=<pnumber_of_ath_to/id_nacl>] [--identity=<path_to/id_nacl>] [--log-level=n] [-l <engine>.prt] [--max-uows=n] [--parse] [> <engine>.rpt]
| Keyword | Description |
|---|---|
| <engine>.sqd | The name of the Apply Engine script file used as input to the parser phase. The recommended file extension for this file in the Multiplatforms environment is sqd. |
| [<parm1> <parm2> …<parmn>] | One or more optional parameters used in the parser phase. See section Substitution Parameters in the Parser Directives section above for an explanation of the two types of parameters supported: Named and Positional. Examples will use the recommend Named parameter syntax. |
| [--fork=<pnumber_of_ath_to/id_nacl>] | Run multiple copies of the engine at once, distributing the input between them intended for high volumn loads only. Precisely recommends discussing the use of this parameter with Support before its use. |
| [--identity=<path_to/id_nacl>] (Non-z/OS only) | Local file system path and file name or AKV url for the NaCl private key to be used by the Engine. On z/OS platforms, both Public and Private Key files are specified at runtime by DD statements. |
| [--log-level=n] (Non-z/OS only) | Diagnostic level of logging, 0=none (not recommended), 2=normal (DEFAULT), 4=warnings, 8=detailed (use only as recommended by Precisely Support and usually in conjunction with "diagnostic" version of product (../bin/dbg). A diagnostic log file named <executible>_<pid>.log will be generated for any log level greater than zero (0). Diagnostic log files may range in size from 2-3 lines to an unlimited number of lines so file system size may need to be monitored. |
| [-l <engine>.prt] | The name of the optional Parser report file created at the start of execution. If this parameter is not specified, no parser report is produced. The suggested file extension for this file in the Multiplatform environments is .prt. |
| [--max-uows=n] (Linux Kafka targets only) | Optional runtime parameter that overrides the default of 100 in-flight units-of-work (UOWs). Some Kafka (and compatible) targets may be unable to manage the default number of UOWs at the same time. That may be due to capacity issues as well as cloud service subscription level. Consider this an additional tuning parameter besides those specified in the Producer Configuration file. Precisely recommends discussing anticipated workload with the team/provider of the target environment and reviewing the recommended producer settings and if applicable, their relationship to subscription service level. Confirm that recommended producer parm value documentation applies to your current subscription levels. |
| [--parse] | Instructs sqdeng to only parse the script and not attempt execution. Use together with -l to only parse and produce a full Parser report. |
| [> <engine>.rpt] | The name of the runtime report file produced during execution of the Apply Engine. If this parameter is not specified, the report output will be written to the screen/terminal in the Multiplatform environment. The suggested file extension for this file in the Multiplatform environments is rpt. |
- When using SQDENG the working directory is a bit more critical because references in the script to the location of Source descriptions, called procedures, etc must be sensitive to their location in the directory hierarchy, ie: ./<directory_name>/<file_name>.
- Azure Key Vault (AKV) based secrets in Connect CDC (SQData) are supported only on Linux platforms.
- AKV requires an Azure Active Directory (AAD) token to be presented to retrieve secrets. SQData retrieves AAD tokens from Azure differently when running on on-prem Linux machines and when running on Azure Linux VM.
- When running on-prem, to retrieve AAD, tenant_id, client_id and client_secret have to be specified in the sqdata_cloud.conf file located in the working directory.
- When running on Azure VM, AAD will retrieved from managed identity. Two types of managed identities are supported:
- If client_id is specified in sqdata_cloud.conf file, then AAD token is retrieved from user managed identity
- If tenant_id, client_id and client_secret are not specified, then AAD token is retrieved from system managed identity.
Example 1
Execute an engine script named DB2TOORA.sqd that will perform near real-time replication of "Sales" application data from a DB2 database captured on a z/OS system named "MVS21" to Oracle tables running on a UNIX system. The source "Host" system and DB2 System ID were passed as parameters because the engine script has been written so that it can be reparsed to process data coming from other systems as well.
sqdata ./ENGINE/DB2TOORA.prc 2>&1 | tee ./ENGINE/DB2TOORA.rpt
Execution of the same engine using SQDENG, might look like this:
sqdeng --script=./ENGINE/DB2TOORA.sqd ENGINE=DB2TOORA HOST=MVS21 SSID=DB2T --list=./ENGINE/DB2TOORA.prt > ./ENGINE/DB2TOORA.rpt