New functionality - Connect_ETL - Latest

Support for higher precision arithmetic expressions with only constants. When the environment value DMX_ARITHMETIC_HIGH_PRECISION is set to any value other than AUTO, a quad precision floating point number will also be used as the intermediate data type instead of a double precision floating point number for arithmetic binary operations (+-*/) with constant operands. This previously only affected arithmetic expressions with fields or a combination of fields and constants. This will provide 33 significant digits of precision instead of 15, but there will be an impact on performance. If DMX_ARITHMETIC_HIGH_PRECISION is set to AUTO, the performance of those binary arithmetic operations will be improved by using a 64-bit or 128-bit integer instead of a floating point number as the intermediate data type for decimal/integer operands when possible.

Task Editor support for variable record length with little-endian prefix. The Task Editor now supports a new file type "Variable record length with 2-byte record prefix (little endian)" and record format "Variable length with 2-byte prefix (little endian)" in the corresponding source and target dialogs.

Mainframe variable maximum record length increased. The maximum record length for the "Mainframe variable record length" file type (MAINFRAME VARIABLE in DTL, MAINFRAMEUNBLOCKEDVARIABLE in SDK) has been increased from 32756 to 32760 bytes. Tasks created using the Task Editor will need to be resaved in order for the new limit to take effect.

Improved performance for high precision arithmetic operations. When the environment variable DMX_ARITHMETIC_HIGH_PRECISION is set to AUTO, high precision integers will be used instead of high precision floating point as the intermediate data type when possible. This mainly affects addition, subtraction, and multiplication arithmetic operations with decimal and binary integer data type operands, including integer constants.

Additional SFTP key exchange support. Connect ETL now supports the following key exchanges for SFTP: diffie-hellman-group14-sha256, diffie-hellman-group16-sha512, and diffie-hellman-group18-sha512

Launch of Connect ETL on Databricks for Azure Cloud. Precisely Connect ETL now supports customers who want to leverage the analytics capabilities of Databricks for Azure Cloud without recalibrating their ETL pipelines. With this release, Connect ETL customers can move their existing ETL jobs to Databricks easily by running data processing on Databricks under the Intelligent Execution (IX) layer and then moving their current non-Databricks ETL workloads to Databricks clusters.

Improved integration with Azure Cloud services. Precisely Connect continues to improve integration with the Azure cloud platform. This release provides integration with Azure Key Vault; customers can manage their secrets in Azure multi-tenant environments, and Connect ETL can securely access secrets from the Vault on-the-fly to establish data connections. This release adds support for Azure Active Directory (AD) authentication, enabling Connect ETL jobs to use Azure AD for authentication.

Connect ETL now provides direct support for Azure storage which enables customers to read/write to Azure storage from on-prem, hybrid, and on-cloud environments.

Connect ETL/Connect for Big Data now supports the ability to configure the degree of parallelism to be passed to the Oracle merge query via a parallel hint, if using the update or update-and-insert target table dispositions.By default, Connect maintains existing behavior and constructs the PARALLEL hint such that the Oracle database server determines the degree of parallelism to use to execute the query. To configure the PARALLEL hint syntax generated by Connect, and therefore the parallelism used by Oracle to execute the merge query, set DMX_ORACLE_PARALLEL_HINT_VALUE=n,

where n can have the following values:

If n > 0, merge queries for Oracle target tables will be generated with the syntax for a PARALLEL hint with n degrees of parallelism

If n = 0, merge queries for Oracle target tables will be generated with the syntax for the NOPARALLEL hint

If n < 0, merge queries for Oracle target tables will be generated without any Oracle hints

Protegrity Data Security Gateway. Connect ETL/Connect for Big Data now supports protecting data using API calls to Protegrity Data Security Gateway. See the Connect ETL help topic, "Protect function", or Install Guide topic, "Connecting to Protegrity Data Security Gateway".

Db2 for Linux, UNIX, and Windows data source support in Precisely Connect Portal. Connect Portal supports Db2 for Linux, UNIX, and Windows (LUW) JDBC data source connections to replicate your captured data changes to Apache Kafka targets. For more information, see the Connect Portal help topic, "Adding New Data Connections."

Databricks databases. Through JDBC connectivity, Connect for Big Data supports Databricks databases as sources and targets for all dispositions. See the Connect ETL help topic, "Connecting to Databricks."

Db2 for IBM i data source support in Precisely Connect Portal. Connect Portal supports Db2 for IBM i JDBC data source connections to replicate your captured data changes to Apache Kafka targets. For more information, see the Connect Portal help topic, "Adding New Data Connections."

Abort on Rejected Records and ENFORCELENGTH parameter for Vertica targets. The "Abort if any record is rejected" parameter configures Connect ETL to abort tasks that update Vertica targets upon rejecting a record. The default behavior to process remaining records remains unchanged. A new Target Database Table parameter Vertica COPY ENFORCELENGTH enables the Vertica COPY command ENFORCELENGTH option when loading Vertica targets. Please see the Connect ETL on-line help topic "Connecting to Vertica Databases" for details.

CyberArk Enterprise Password Vault. Connect ETL/Connect for Big Data now supports retrieving passwords stored in a CyberArk vault to authenticate connections to servers, databases, message queues, and remote files. See the Connect ETL help topic, "Task Editor", or Install Guide topic, "Connecting to CyberArk Enterprise Password Vault".

Azure Synapse Analytics (formerly SQL Data Warehouse). Through JDBC connectivity, Connect for Big Data supports Azure Synapse Analytics as sources and targets for all dispositions. See the Connect ETL help topic, "Connecting to Azure Synapse Analytics."

Snowflake databases. Through JDBC connectivity, Connect for Big Data supports Snowflake databases as sources and targets for all dispositions. See the Connect ETL help topic, "Connecting to Snowflake."

For more information about how to use Connect Portal, see the Connect Portal help available from the application. To get started with Precisely Connect Portal, you need to install Connect ETL and MIMIX Share (CDC).

Oracle and SQL Server targets through JDBC connections supported in Hadoop Single Node Execution - Connect ETL offers the ability to move data from Hadoop distributed file system and database sources to Oracle and SQL Server targets in a Single Node Execution Connect ETL job. Please see the on-line help topics, Defining Connect for Big Data ETL Sources and Targets, Connecting to JDBC Sources and Targets, and Running a job from the command prompt for more information.

Separate software and license packages for Connect for Big Data. Previously, customer-specific Connect for Big Data installation packages with embedded license keys were generated for RPM, Cloudera parcel, and Ambari package installations. Now, the software and your license keys are provided in separate packages. This allows you to apply a new license key to all cluster nodes without reinstalling Connect for Big Data. Refer to the Installation Guide.

Abort on Rejected Records to Kafka and MapR streams. The new "Abort if any record is rejected" parameter configures Connect ETL to abort tasks that update Kafka targets upon rejecting a record. The default behavior to process remaining records remains unchanged.

Distributed Parallel Data Ingestion (First customer shipment). Connect Portal offers distributed parallel ingestion of large database sources for enabled jobs to Hadoop targets, which enables parallelized map jobs to any number of Hadoop targets (HDFS, Hive, or HCatalog). With distributed parallel ingestion, DMX automatically splits tables across mappers to optimize load performance based on table and cluster size. For more details, see the DMX help topic, "High-speed parallel ingestion of large database sources to Hadoop."

Connect ETL Change Data Capture for DB2 z/OS. Added READ authority information and missing SYSIBM table names for DB2 Catalog Tables in DB2GRANT JCL . Refer to the "Identify/Authorize z/ OS Users and Started Tasks" section in the DB2 z/OS Change Data Capture Reference.

Support for COBOL-IT line sequential files on SunOS SPARC 64-bit. Connect ETL/Connect for Big Data provides support for COBOL-IT line sequential files. See the Connect ETL/Connect for Big Data help topic, "Installing support for COBOL-IT."

• Checking whether a column is the first or last one in the source table, in the predicate of expression_construction transformation rules
• Creating target files in subdirectories named for each source table
• Customizing the file extension of target files
• Customizing the file extension of exported DTL task files

See the Connect ETL/Connect for Big Data help topic, "DMXDFNL configuration file."

Runtime field-level data lineage. Connect ETL/Connect for Big Data offers a runtime data lineage solution that enables you to track field-level metadata from source to target through the Connect ETL/Connect for Big Data task and job processes that transform it. As part of your overall data governance strategy, you can leverage the generated lineage for debugging, auditing, and regulatory reporting. You can further extend these capabilities through end-to-end visualization by securely publishing the lineage to Cloudera Navigator as a compliance-ready solution. See the Connect ETL help topic, Connect ETL/Connect for Big Data runtime data lineage. Connect ETL/Connect for Big Data lineage is connected to all other data lineage in the cluster, which offers a full view of what is happening to the data within and without Connect ETL/Connect for Big Data.

VSAM source support for Connect CDC for z/OS. In addition to DB2 for z/OS sources, Connect CDC for z/OS supports VSAM data sets as sources. Changes written to VSAM data sets can be captured and written to target files and applied to Hive target database tables through generated downstream Connect for Big Data tasks. See the Connect ETL/Connect for Big Data help topic, "Connect Change Data Capture."

• Additional array selection option, which enables mapping all elements of any array field within a chain of fields into separate fields.
• Ability to create tables in which the hierarchy of composite fields is either flattened or maintained.

See the Connect help topics, "Array Element Selection dialog" and "Target Database Table dialog."

Apache Impala databases. Through JDBC connectivity, Connect for Big Data supports Impala databases as sources and targets when running on the ETL server/edge node, in the cluster, and on a framework-determined node in the cluster. See the Connect help topic, "Connecting to Impala."

• Additional array selection option, which enables mapping all elements of any array field within a chain of fields into separate fields.
• Ability to create tables in which the hierarchy of composite fields is either flattened or maintained.

See the Connect help topics, "Array Element Selection dialog" and "Target Database Table dialog."

Connect Enterprise architecture (First customer shipment). In addition to the classic Connect client-server architecture in which Connect ETL/Connect for Big Data jobs and tasks are run, Connect ETL also supports an enterprise web-server architecture, which extends Connect ETL processing power and currently supports running Connect Portal projects through a browser-based user interface (UI).

See the Connect ETL help topic, "Connect ETL Enterprise architecture."

Connect Portal UI. Connect Portal UI is a secure, high-speed, browser-based application from which you collectively extract, filter, transfer, and copy large volumes of data from database sources to target database management systems (DBMS) or file systems (including HDFS, S3, and local).

As an extension of Connect ETL/, Connect Portal UI enables you to automatically run data transfers in bulk rather than manually creating dozens, hundreds, or even thousands of individual transfer tasks, one task for each table in the schema you want to transfer. Instead, Connect Portal UI gives you the tools to build collections of data transfer tasks called data flows. You group data flows in projects. When a project is executed, each customized data flow automatically transfers your data in bulk to a specified target database or file system.

For more information about how to use Connect Portal UI, access the online help, which is available from the Help menu in Connect Portal UI.

Enhanced Connect ETL/Connect for Big Data installation. The Connect ETL/Connect for Big Data installation process includes component-based options, which enable standard, full, classic, and custom installations. From among these options, you can tailor the installation to meet your requirements; for example, you can install a client-server configuration for running Connect ETL/Connect for Big Data jobs and tasks through the Job and Task Editors, or you can install a web-server configuration for running Connect Portal projects through the Connect Portal UI, or you can install both. See the Connect ETL/Connect for Big Data help topics, "Connect ETL/Connect for Big Data installation component options" and "Connect ETL/Connect for Big Data Enterprise architecture."

Target support for Hive complex types. Connect for Big Data supports writing elementary, composite, and array fields from mainframe and open system database table and file sources to Hive target structures and arrays. See the Connect ETL/Connect for Big Data help topic, "Target Database Table dialog."

Hive merge statement support. Connect for Big Data provides Hive merge statement support through JDBC for Hive target database tables that are configured to support ACID and that are specified as having the upsert or update disposition. See the Connect ETL/Connect for Big Data help topic, "Hive merge statement support for upsert and update dispositions."

User-defined SQL as lookup source. In addition to supporting entire database tables as lookup sources, Connect ETL/Connect for Big Data supports user-defined SQL statements as lookup sources. See the Connect ETL/Connect for Big Data help topic, "Lookup Source Database Table dialog."

DMX Change Data Capture. DMX Change Data Capture (CDC) is a new add-on product that keeps data on a mainframe source system in sync with Hadoop without overloading networks and incurring a high MIPS cost on the mainframe. Connect CDC for z/OS detects changes in the source system and either applies or writes the changed data to a specified Hadoop target. Currently, DB2 for z/OS is supported as source and Hive and HDFS are supported as targets. See the Connect ETL/Connect for Big Data help topic, "DMX Change Data Capture."

See the Connect ETL/Connect for Big Data help topic, "Developing Intelligent Execution Jobs."

Connect Portal support for Amazon S3 and Amazon Redshift. Connect Portal supports Amazon S3 as target and Amazon Redshift as source and target. See the Connect ETL/Connect for Big Data help topics, "Connect Portal," "DMXDFNL configuration file," and "Connections object: amazon_s3 member."

Enhanced Hive support. Connect for Big Data supports Hive database table lookup sources, the Boolean data type for Hive JDBC source and target connections, and updates applicable to the integration with Apache Ranger. See the Connect for Big Data help topics, "Connecting to Hive data warehouses," "Conversion between Hive data types and Connect for Big Data data types," and "Apache Sentry and Apache Ranger authorization."

Connect ETL/Connect for Big Data job progress monitoring and reporting. Support for real-time progress monitoring of jobs and their subjobs and tasks and report generation on this progress for jobs run in both Connect ETL and in MapReduce/Spark. See the Connect ETL/Connect for Big Data help topic, "Connect ETL/Connect for Big Data progress monitor."

Key break processing functions. Connect ETL/Connect for Big Data supports key break processing functions for sort, merge, and aggregate tasks. While cross-record processing can be achieved with user defined values, key break processing functions provide a simplified means of enabling cross-record processing with sort keys, merge keys, or group by fields, which are specified as arguments in these functions. Depending on the executed function, the returned value can represent changes in argument values or can be counters or running totals. See the Connect ETL/Connect for Big Data help topic, "Key break processing functions."

Support for Spark cluster deploy mode. Connect for Big Data supports running jobs on Spark in cluster deploy mode as well as in client deploy mode. See the Connect ETL/Connect for Big Data help topic, "Connect for Big Data Configuration Settings."

Access method support in Connect Portal, including Hive JDBC. Users can now specify the access method for each database connection in Connect Portal. This includes the JDBC access method for Hive connections, which is now recommended. See the definition of the "access_method" parameter in the Help topic "Connections object: database member".

Hive target statistics. Upon loading to Hive target tables, Connect for Big Data can now analyze table-level and column-level statistics for the target table, which optimizes subsequent Hive query performance. See the Connect ETL/Connect for Big Data help topic, "Connecting to Hive data warehouses."

DTL metadata export. Connect ETL/Connect for Big Data supports the export of metadata, which includes values, conditions, layouts, connections, and collating sequences, to DTL command options files. See the Connect ETL/Connect for Big Data help topic, "Export Connect ETL/Connect for Big Data job and task files into DTL command options files."

Google Cloud Storage sources and targets. Connect ETL/Connect for Big Data supports direct connections to Google Cloud Storage buckets for reading and writing. See the Connect ETL/Connect for Big Data help topic, "Connecting to Google Cloud Storage from Connect ETL/Connect for Big Data."

Ability to create target tables in Hive at design-time. It is now possible to create target tables in Hive at design-time for access via JDBC. This can be achieved using the "Create Database Table" dialog. See the Connect ETL/Connect for Big Data help topic, "Create Database Table dialog."

Record number in warning message. When a data issue is encountered in processing records, Connect ETL/Connect for Big Data provides you with the ability to view the corresponding record number, source number, and join side (if applicable) in the generated warning message. See the Connect ETL/Connect for Big Data help topic, "Connect ETL/Connect for Big Data debugging."

Distributed Hive sources and targets. Through JDBC connectivity, Connect for Big Data supports reading from Hive sources and writing to Hive targets in the cluster. In particular, this allows Parquetbacked Hive tables, which could only be read and written on the edge node previously, to be read and written by jobs run in the cluster. See the Connect ETL/Connect for Big Data help topic, "Connecting to Hive data warehouses."

Integrated workflow. Users can now specify where to run individual task and subjob components of a Connect for Big Data job - on a cluster, single node of a cluster, or single server. This orchestration capability allows a single job to perform both data ingestion from the edge node and distributed processing in Spark and MapReduce. See "Integrated workflow" in the Connect ETL/Connect for Big Data help.

Support for COBOL-IT line sequential files. Connect ETL/Connect for Big Data provides support for COBOL-IT line sequential files. See the Connect ETL/Connect for Big Data help topic, "Installing support for COBOL-IT."

Connect Portal (DMXDFNL) table creation support. Leveraging customizable metadata translation, DMXDFNL can now automatically create tables for Hive/HCatalog targets based on source tables from the following databases: Oracle, DB2 for z/OS, Netezza, Microsoft SQL Server, and Teradata. Generated table definitions (DDL/CREATE TABLE statements) can be exported to files on disk using the ddl_export_dir configuration parameter. See the Connect ETL/Connect for Big Data help topics: "Connect Portal" and "Subunit alias object: table_creation member."

DMXDFNL inflight transformation support. DMXDFNL can now map sources to targets, transform columns based upon their properties, and map source expressions to target fields dynamically. Transformation rules, which are referenced in the data transfer task, are programmatically defined at the root level of the DMXDFNL configuration file. See the Connect ETL/Connect for Big Data help topics: "Connect Portal," "DMXDFNL configuration file," and "Subunit alias object: data_transfer member."

• When output to a Linux/UNIX terminal, log messages now display in color: errors display in red and warnings display in yellow.

• Tasks no longer time out by default. A timeout can be specified globally with the root-level default_timeout_minutes configuration parameter or at the subunit_order level with the timeout_minutes parameter.

• The no_execute configuration parameter now supports disabling the execution of one or more task types.
• DMXDFNL configuration file updates include the following:
  • Tasks are separated into table_creation, data_transfer, and existing_dtl categories in each subunit.
  • Sources and targets are defined at the root level.

The previous configuration styles are deprecated, but remain supported. See the Connect ETL/Connect for Big Data help topic, "DMXDFNL configuration file."

Google Cloud Storage sources and targets. Connect ETL/Connect for Big Data supports reading from and writing to Google Cloud Storage buckets in and out of Hadoop. See the Connect ETL/Connect for Big Data help topic "Connecting to Google Cloud Storage from Connect ETL/Connect for Big Data."

MapR Streams sources and targets (GA). Connect for Big Data supports MapR Streams as a message queue source when running from an edge or single cluster node, and as a message queue target when running from an edge node, single cluster node, or MapR cluster. Kerberos is supported for connectivity to MapR Streams. See "MapR Streams" in the Connect ETL/Connect for Big Data help.

DB2 ODBC driver support for Windows 32-bit. Connect ETL/Connect for Big Data now provides DB2 ODBC driver support for Windows 32-bit installations in addition to the support for UNIX/Linux installations. See the Connect ETL/Connect for Big Data help topic, "Connecting to DB2 databases."

Connect ETL/Connect for Big Data Report enhancements. Connect ETL/Connect for Big Data supports the generation of Connect ETL/Connect for Big Data job and task reports in machine processable, comma separated values (csv) format. Through command-line dmxreport invocation and wildcard support for input job and task files, you can generate data from an unlimited number of job definitions, interpret the data with data analysis tools, and load the data into databases and into other metadata repositories. See the Connect ETL/Connect for Big Data help topic, "dmxreport: Generating Connect ETL/Connect for Big Data job and task reports."

Hive targets via JDBC. Hive targets can now be written via a JDBC connection, allowing for easier configuration, particularly when running on a single cluster node. See "Connecting to Hive data warehouses" in the Connect ETL/Connect for Big Data help.

Hive targets support truncate and insert. Hive targets now support the "Truncate table and insert rows" disposition. See "Connecting to Hive data warehouses" in the Connect ETL/Connect for Big Data help.

Repository passwords. Connect ETL/Connect for Big Data now supports storage of password and secret key values in a secure repository under a user-provided name. This name can subsequently be referenced while creating Connect ETL/Connect for Big Data tasks or Connect Portal jobs. See "Repository passwords" in the Connect ETL/Connect for Big Data help.

MapR Streams sources and targets (Pre-release). Connect ETL/Connect for Big Data supports MapR Streams as a message queue source when running from an edge or single cluster node, and as a message queue target when running from an edge node, single cluster node, or MapR cluster. Kerberos support is not yet available for connectivity to MapR Streams. See "MapR Streams" in the Connect ETL/Connect for Big Data help.

Support for international character sets for data and metadata. Connect ETL/Connect for Big Data now enhances support for globalization through ICU character sets and multi-byte COBOL copybooks. See the Connect ETL/Connect for Big Data help topics "Data encoding" and "External metadata in a COBOL copybook."

DMXMMSRT supports NULL decimal fields. The DMXMMSRT utility now supports an option to treat empty or whitespace-only decimal fields (except Packed Decimal) as NULL. See "Mainframe migration sort component DMXMMSRT" in the Connect ETL/Connect for Big Data help.