Streaming Data Store Output - Latest

Data360 DQ+ Help

Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 DQ+
Version
Latest
Language
English
Product name
Data360 DQ+
Title
Data360 DQ+ Help
Copyright
2024
First publish date
2016
Last updated
2024-10-17
Published on
2024-10-17T10:11:12.223717

The Streaming Data Store Output node outputs streaming data to a data store.

This node performs a similar function to the Data Store Output node, in that it forms an Analysis end point, where you can choose the data store in which you want to hold the fields manipulated by an Analysis. In the case of the Streaming Data Store Output node, you can only choose a data store which has a Store Repository Type of Kafka. For example, you may want to store the data in a Kafka topic on AWS.

For more information on configuring external data stores, see Creating an external data store.

Properties

Display Name

Specify the name of the node that is displayed on the Analysis Designer canvas.

The default value is Streaming Data Store Output.

Data Store to Read

Select a streaming data store.

You can only select a data store which has a Store Repository Type of Kafka and where the option This channel will be used to write to Kafka is selected.

A value is required for this property.

Value Field

Select an incoming string field.

A value is required for this property.

Key Field

Optionally select an incoming key field.

If you select a Key Field, this field will be displayed in the spreadsheet on the Analysis page, along with the Value Field.

Output Mode

Select an output mode. Choose from:

  • Append - Adds new rows. Append mode supports queries where the rows that are added are never going to change, for example, select, filter or join queries.
  • Update - Outputs rows that were updated since the last trigger.
  • Complete - Outputs all rows after every trigger. Complete mode supports aggregation queries.

The default value is Append.

The following table lists the different streaming queries that are supported by each output mode:

Query type

Supported output mode

Queries with aggregation
  • All output modes are supported for aggregations that use watermarking. For aggregations that do not use watermarking, only the Update and Complete output modes are supported.
  • A Streaming SQL node with aggregation will not work if a subsequent Convert to Micro Batch node is in Append mode.
Queries with joins
  • If a Convert to Micro Batch node follows a Streaming Join node, the Output Mode property must be set to Append.
Other queries
  • Only the Append and Update output modes are supported for queries where the data is not aggregated.