Build schema manually - Data360_DQ+ - Latest

Add the initial struct

Starting with a blank schema, you should first add a Struct field. This field will contain all the fields of your schema content.

1. Click Add.
2. Select Field from the drop down list. The Add Transformation dialog is displayed.
3. Leave Source Name blank.
4. For Source Data Type, choose "Struct".
5. Click Save.

Add fields

Next you need to add fields and arrays, depending on the structure of your content.

1. Select the Struct you created above.
2. Click Add.
3. Select Field from the drop down list. The Add Transformation dialog is displayed.
4. Enter the Source Name.
5. Select the Source Data Type from the drop down list. For information about the available types, see Source Data Type.
6. Using the Output Transform Action property, select whether to "Copy" the source to the output, or "Ignore" the field.
7. If the Source Data Type is "Number", use the Output Field Type property to define the number type of the output field. The available options are "Big Integer", "Decimal", "Floating Point", "Integer".
8. When you selected the Source Data Type, the Output Field Name was added, with the same value.
9. Click Save. The Add Transformation dialog closes, and the Edit Transformations window is updated, with the new field added as part of the Struct.
10. Click OK to close the schema and transformations builder.
11. Click Accept Changes.

Edit fields

To edit an existing field:

1. Select the corresponding row for the field in the table.
2. Click Edit in the header bar. The Edit field Transformation dialog is displayed.
3. Modify the transformation from the source field to its output by changing the value of one or more options. Note that some options might be disabled.
4. Click Save.
5. Click OK in the Edit Transformations window to confirm your changes.

Transformation options

Source Name

The name of the field.

Source Data Type

The Schema and Transformation Builder supports the following data types.

• Array - can be an array of elements of any of the other supported types, except Map. Arrays of mixed type are not supported. The transformation builder can handle arrays in a number of ways. You can copy an array, or explode the contents, choosing to include or exclude null values. For more information, see Array handling in the transformation builder.
• Boolean
• Date
• DateTime
• Map
• Number - can be Big Integer, Decimal, Floating, or Integer.
• String
• Struct

Notes:

Output Transform Action

The available Output Transform Actions depend on the Source Data Type value.

When the Source Data Type is an array, you can Copy the field into the output as an array, Ignore the field and exclude it from the output, or choose an Explode option to generate multiple output rows. For more information, see Array handling in the transformation builder.

When the Source Data Type is a non-array value, you can choose to Copy the field into the output, or Ignore the field and exclude it from the output.

Related Field

This option is available only when the Source Data Type is an array, and the Output Transform Action is "Pos Explode" or "Pos Explode Outer".

Output Field Type

You can only select an Output Field Type when the Source Data Type is "Number" or "Array (Number)". Use this option to provide a specific numeric type for an output field.

Output Field Name

The name of the field in the output.