Fixes that change product behavior - Connect_ETL - Latest

• and a custom task exists, all tasks of the job run on the local ETL server/edge node unless otherwise specified by the environment variable DMX_HADOOP_ON_VALIDATION_FAILURE
• extended tasks or groups of directly connected extended tasks run on the cluster with all Connect for Big Data tasks when they have piped inputs from previous Connect for Big Data tasks and piped outputs to following Connect for Big Data tasks.

• custom/extended tasks that have no inputs/outputs run on the local ETL server/edge node and all Connect for Big Data tasks run on the cluster.

• both custom and extended tasks or groups of directly connected custom/extended tasks run on the cluster with all Connect for Big Data tasks when they have piped inputs from previous Connect for Big Data tasks and piped outputs to following Connect for Big Data tasks

In the Connect ETL/Connect for Big Data Job Editor, a group of tasks that share data in the form of pipes, previous/next task pairs, or direct data flows is visible through the "Show Execution Units" context menu item and View menu item

In the Connect ETL/Connect for Big Data Job Editor, a group of tasks that share data in the form of pipes, previous/next task pairs, or direct data flows is visible through the "Show

Restartability Units" context menu item and View menu item

The Connect ETL/Connect for Big Data Task Editor no longer displays errors for invalid metadata files when you select a file in the External Metadata dialog without specifying a metadata type.

The Connect ETL/Connect for Big Data Task Editor detects the metadata type and prompts you to switch to this type when you specify a metadata file name with an incompatible metadata type.

The Connect ETL/Connect for Big Data Task Editor no longer detects the metadata type nor does it prompt you to switch to this type when you specify a metadata file name with an incompatible metadata type. Connect ETL/Connect for Big Data generates the errors that result from parsing the metadata file with the wrong metadata type

Version number is 27. For existing ODBC data source configurations, ensure that the version numbers in the DataDirect driver file names, which are listed in the driver properties section in the odbcinst.ini and odbc.ini files, are updated from 26 to 27.

Tasks that map to COBOL copybooks containing unsigned packed decimal data. Unsigned packed decimal fields are indicated by a USAGE clause of COMP-3 or PACKED-DECIMAL and a PICTURE (PIC) clause that does not contain an "S" (sign).

Unsigned packed decimal fields are now treated as unsigned. Negative values are not allowed and will cause validation warnings such as CVROP, CVNEGTOUNSIGN or NEG2USEX. Valid (positive) values in unsigned packed decimal prefer the "F" sign nibble rather than "C" on output. See the Connect ETL/Connect for Big Data help topic, "Decimal, packed, unsigned number data format."

The actual number of elements in the input record is output to the variable length array. A task created through the Connect ETL/Connect for Big Data Editor must be resaved to result in the correct behavior.

Connect ETL/Connect for Big Data tasks connecting to HDFS filesystems, where the NameNode hostname specified in the task's HDFS connection does not match the NameNode hostname or HA (High Availability) service name detected at runtime.

The task fails with the following messages:

DMExpress : (HDFSCNNF) HDFS connection to Hadoop namenode (<namenode_host>) failed due to the following reason:

(HDFSNNM) server name "<namenode_host>" specified for the connection does not match the namenode host name "<hdfs_server>" in your Hadoop configuration

The task no longer fails, and the default filesystem (specified by the Hadoop configuration property "fs.defaultFS" or "fs.default.name") takes precedence over the value specified in the task. If the filesystem is an HDFS-compatible filesystem other than HDFS, no messages are issued. If the filesystem is HDFS, the following informational message is issued without affecting the successful completion of the task:

DMExpress : (HDFSNNM) an HDFS connection specifies a server name that does not match the namenode hostname "<hdfs_server>" specified in your Hadoop configuration; the value specified in the Hadoop configuration will be used

dmx_connector_FieldProperties now has an additional field "int m_usageIndicator" for indicating field usage. Existing connectors will need to be modified to initialize the new field.

FieldProperties now has an additional private data member "usageIndicator" for indicating field usage. There is also an additional member function "getUsageIndicator()" to return its value and an additional constructor to set its value.

Connect ETL/Connect for Big Data used the following Connect ETL/Connect for Big Data Kerberos environment variables to manage Kerberos tickets:

DMX_HADOOP_KERBEROS_PRINCIPAL, DMX_HADOOP_KERBEROS_KEYTAB, and DMX_HADOOP_KERBEROS_CACHE

With the introduction of Kerberos authentication for Teradata databases, Connect ETL/Connect for Big Data Kerberos environment variables are not Hadoop specific. The environment variables are renamed as follows: DMX_KERBEROS_PRINCIPAL, DMX_KERBEROS_KEYTAB, and DMX_KERBEROS_CACHE. In addition, the following new environment variable is introduced: DMX_KERBEROS_CACHE_PREFIX. Backward compatibility for the deprecated environment variables is retained. See the Connect ETL/Connect for Big Data help topic, "Kerberos Authentication in Connect ETL/Connect for Big Data jobs."

Record is rejected and sent to rejected records log file. Connect ETL/Connect for Big Data issues a warning message. Additional messages may be listed in the rejected records exception messages log file.

Record is rejected and sent to rejected records log file. Connect ETL/Connect for Big Data issues a warning message. Additional messages may be listed in the rejected records exception messages log file.

Record is rejected and sent to rejected records log file. Connect ETL/Connect for Big Data issues a warning message. Additional messages may be listed in the rejected records exception messages log file.

Record is rejected and sent to rejected records log file. Connect ETL/Connect for Big Data issues a warning message. Additional messages may be listed in the rejected records exception messages log file.

For new tasks or tasks resaved in the current release of Connect ETL/Connect for Big Data, the default extraction format of Teradata character data changes. For Teradata sources that do not have any column defined as Unicode, character data will be extracted in the default Teradata client character set and interpreted in the system locale (instead of ASCII). For Teradata sources that have one or more columns defined as Unicode, character data will be extracted as UTF-8.

Tasks that are not resaved in the current release of the Task Editor will not change. To resave affected tasks and maintain existing behavior,

select the option to keep existing behavior when

prompted at task open, or when running the Connect ETL/Connect for Big Data Application Upgrade utility

(see the Connect ETL/Connect for Big Data Application Upgrade help topic for

details). Alternatively, manually change each character column to extract in Locale encoding and treat as ASCII

Connect for Big Data calculated the input datasize to mappers/reducers based on the HDFS input split size and the number of mappers/reducers.

New options, dmx.map.datasize.useHDFSsplits and dmx.reduce.datasize.useHDFSsplits, when false (default), allow the user to specify the datasize using options dmx.map.datasize and dmx.reduce.datasize to eliminate the need to query the namenode for the input split size by each mapper/reducer. See the Connect for Big Data Sort Edition User Guide for details.

Vertica load validation when a Connect ETL/Connect for Big Data CHAR field with more characters is mapped to a Vertica CHAR column with less.

Vertica load validation when a Connect ETL/Connect for Big Data EN field is mapped to a Vertica INTEGER column with data that forces a numeric overflow.

Record is rejected with a warning and added to the rejected records log file (for Vertica 6 and later).

Vertica load validation when a Connect ETL/Connect for Big Data EN field is mapped to a Vertica NUMERIC column with data that exceeds precision.

Vertica load validation when a Connect ETL/Connect for Big Data EN field is mapped to a Vertica NUMERIC column with data that exceeds scale.

Record is rejected with a warning and added to the rejected records log file (for Vertica 6 and later).

These features are now categorized under the product licensing feature "Advanced Text Processing". See "Licensing" in the Connect ETL/Connect for Big Data help.

Please contact dmlicense@precisely.com if you are currently licensed for the Advanced Data Management license package and using any of these functions. You will need new license keys that enable the Advanced Text Processing feature in order to use these text functions.

Filters that compare a delimited character field with a string shorter than the field length in a non-ASCII locale.

The shorter string is incorrectly not padded to the length of the field with the pad byte.

Tasks with a Teradata target that were saved with Connect ETL/Connect for Big Data version 4.7.12.10 or lower.

Task aborts with the following message: (DBMSABM) abort if error is mandatory for the "Teradata" DBMS.

Each elementary field inside the composite/array field will be created separately.

For example, if a fixed position composite field was being reformatted into a delimited layout:

Input:

01312008abc

Output before the change:

01312008,abc

Output after the change:

01,31,2007,abc

In the Reformat Target Layout dialog, use the "Set target format" option to turn the composite/array field into a single field. Consider using enclosing characters for the target file if you need to reuse the target layout in the next tasks. See the topic "Target File dialog" in the online help to learn more about using enclosing characters.

Tasks with an encoding specified for fields in a record layout which is associated with a source that has a different encoding. Tasks with an encoding specified for fields in a reformat which is applied to a target that has a different encoding.

Field level encoding would determine the field encoding and override file level encoding.

File level encoding will determine the field encoding and override field level encoding.

When field level encoding is desired, leave the file level encoding unspecified.

The Connect service used for communication between the Connect ETL/Connect for Big Data client and the Connect ETL/Connect for Big Data server.

The Connect Service only used ports for the port mapper and an arbitrary port provided by the port mapper.

The Connect Service requires the use of a new port, 32636, in addition to the previously used ports. This new port needs to be accessible through any firewalls and/or security policies present between the systems running the Connect ETL/Connect for Big Data client and the system running the Connect ETL/Connect for Big Data server.

Values are collated and compared according to the Unicode collation algorithm found at http://www.unicode.org/unicode/reports/tr10/using a collation strength of three levels for comparisons.

Replace IsValidDate(datetime) with the expression IfThenElse(datetime=NULL, NULL, IsValidDate(datetime)) in your tasks.

Any tasks that perform transformations that are impacted by nullability. For example, tasks with source database tables, target database tables, comparisons with empty strings, join fields, new values, etc.

NULL Handling options for a data type in the Task Settings impacted nullability attributes for fields or values and changed fields and values of that data type to be nullable, if checked. When the NULL Handling settings in a task were checked for a data type, all empty fields or values of that data type in the task would be treated as NULL. When the NULL Handling settings in a task were not checked for a data type, all empty values of that data type in the task would be treated as NULL or NOT NULL depending on the nullability attribute of the individual field.

Nullability attributes of fields and values are no longer overridden by NULL Handling options in Task Settings. Only fields whose nullability setting is set to "Use Task Setting" are impacted by NULL Handling options in Task Settings. Nullability attributes of fields are determined by the field's settings defined at source field's origin, which may have been defined in a preceding task if the field comes from linked external metadata. Nullability settings of new values are derived from the nullability attributes of the source fields used to create the value. All constants, except the constant NULL, are considered not nullable.

Resaving your tasks in your newest release may correct all of the problems.

Alternatively, change the nullability attributes to "Indicated by empty field" for the target field in a task preceding the task that uses the field in a transformation that relies on NULL behavior. If you have no preceding task, you will need to add a conditional value to check if the field or value is empty and if so, use the NULL constant, else use the field or value.