New in 3.14.0 - Data360_Analyze - Latest

Data Repository feature

The new "Data Repository" feature allows you to manage and permission access to data on file systems through the Data360 Analyze Directory. You can upload and download data files in the Directory, ready for use in your data flows and nodes.

The data that you access could be stored in the Data360 Analyze local file system, or on a shared drive. Data files are stored in data collections in the Directory. Data collections provide a view into a location on a mounted file system, and they are the location within Data360 Analyze where you can upload, download and store data files.

For the server edition of the product, using the Data Repository in combination with permissions and execution sandboxing enables you to use Data360 Analyze as a "Shared Service", where you have a single installation that is accessible to multiple business teams. You can assign permissions to resources such as data files, data flows and nodes to ensure that each team can only access the assets that belong to them.

For Desktop deployments, access to the file system has changed with the introduction of the Data Repository feature. Previously, Data360 Analyze allowed access to the entire file system. The Data Repository feature limits access to the drive that Data360 Analyze is installed on. It is accessed through the data collection named after the drive. To access other drives it is necessary to create data connections and data collections that map to them.

Changes to node properties with the new Data Repository feature:

• Data folder and data file property types have been added to support the ability to select files and folders within data collections. Note that these property values evaluate to a resource path, which cannot currently be resolved to a file system location in user-written Jython-based nodes (for example, Transform and Lookup) or Java nodes.

• A property ls.lae.legacyFilenameReferenceSupport has been introduced to allow for the old filename property types to continue to work for existing installations and upgrades and allow you to select file system locations under the old share directory controlled by the ls.lae.shareRootDir property. The legacyFilenameReferenceSupport property will be on by default on desktop and upgrade, but switched off by default on a fresh server installation.

Execution sandboxing updates

When using the isolated docker execution containers, nodes can only access data that is available from within data collections that the running user has access to, and their temporary execution data location. For access to additional file system locations, including mapped drives, system administrators should configure data connections, mount points and data collections accordingly.

On Windows, if data connections are setup to locations that are not on the C:\ drive, the repositoryTargetMappings.json mappings file must be configured to allow access to the underlying data connection location from the execution containers. Previously, mapping was done using this user defined mount mappings property: ls.brain.server.docker.mounts. While that property can still be used, the targets specified in the property will be mapped into all containers.

Node audit logs

Audit log files are now created for all nodes that read or write to the file system, except nodes that have been deprecated. These audit log files are saved at: <siteDirectory>/logs/nodes/<date - CCYY-MM-DD format>/<username>/<dataFlow>/<runId>/

The node log will be referenced in the lae-audit.log in the nodeProcessed audit code. The logs are segregated by date to allow for easier archiving. The audit logs produced by nodes can be switched off by adding this line to the cust.prop file: ls.brain.node.auditNodeFileAccess=false

Database Profiler node

The Database Profiler node now supports profiling of Azure Synapse databases.

Upload Tableau node

Added a new asJob property to the Upload Tableau node. Setting this property to "True" allows the file upload to be performed asynchronously. The node will output the jobID that can be used in a subsequent node to check the upload status.

SharePoint 365 nodes

A new way of filtering has been added in the Download for SharePoint 365, List for SharePoint 365 and Delete for SharePoint 365 nodes. This allows for file or folder retrieval based on values entered in the new properties FileName and FileNameMatchingCriteria.

The file or folder that is being searched for must start with the value that is entered. All SharePoint 365 Nodes nodes now have three new output fields which allow for a connection between SharePoint 365 nodes using field properties from these output pins:

The following properties are deprecated:

Publish to DQ+ node

The previously experimental Publish to DQ+ node is now fully supported.

New Proxy and Retry connection properties have been added to the node.

ADLS nodes

The ADLS nodes have been updated to support Azure Data Lake Storage Gen 2. As part of this change, these nodes will no longer support Gen 1 storage accounts. The following announcement by Microsoft instructs users of Gen 1 to migrate to Gen 2: https://azure.microsoft.com/en-us/updates/action-required-switch-to-azure-data-lake-storage-gen2-by-29-february-2024/

Deprecated nodes

These superseded nodes have now been deprecated:

Usability enhancements

Error messaging improvements

You can configure the text for all unexpected error messages returned by REST APIs to be redacted for security purposes. The error codes for the errors will still be reported, but the error messages themselves will be replaced by a message indicating to check the application logs.

To enable redaction of all error messages, add this line in the <site-dir>/cust.prop file: ls.framework.ls.web.rest.restErrorMessageRedactorName="allText"

Restart the web application for the change to take effect. Note that by enabling this, some configuration features may be affected and require the ability to consult the application logs to diagnose issues, for example authentication configuration.

Performance improvements

Improved the performance of the user interface when using the application over an extended period of time.

Third party library updates

A number of third party libraries have been updated to take advantage of any security and feature updates, including:

Breaking changes

Other changes

• The Output Tableau (Deprecated) node has been removed from the product. The newer Output Tableau and Upload Tableau nodes provide similar functionality.
• Added a keep alive page at /html/keepalive.html which renders the text "alive" and can be used to ensure that the web application is running.
• JDBC nodes now use Column Labels (or Titles) for the metadata. Previously they used Column Names. This is to align with the JDBC specification, which states that column labels should be used rather than column names. Note that this change was made in 3.11.0 but was not documented in the release notes at the time.

Corrected issues