Flat and Hierarchical Data - 23.1

Spectrum Dataflow Designer Guide

Version
23.1
Language
English
Product name
Spectrum Technology Platform
Title
Spectrum Dataflow Designer Guide
First publish date
2007
Last updated
2024-05-09
Published on
2024-05-09T23:01:03.226155

Spectrum Technology Platform supports flat data and hierarchical data. In general you can use either flat or hierarchical data as input and output for a flow. A few stages in Spectrum Spatial Routing require data to be in a hierarchical format.

Flat Data

Flat data consists of records, one on each line, and fields in each record. Fields are delimited by a specific character or positioned in a defined location on the line. For example, this is flat data with comma-delimited fields:

Sam,43,United States
Jeff,32,Canada
Mary,61,Ireland

To read flat data into a flow, you can use the Read from File, Read from DB, or Input stages. To write flat data output from a flow, you can use the Write to File, Write to DB, or Output stages.

Hierarchical Data

Hierarchical data is a tree-like structure with data elements that have parent-child relationships. Spectrum Technology Platform can read and write hierarchical data in XML and Variable Format File format. For example, this shows hierarchical data in XML:

<customers>
	<customer>
		<name>Sam</name>
		<age>43</age>
		<country>United States</country>
	</customer>
	<customer>
		<name>Jeff</name>
		<age>32</age>
		<country>Canada</country>
	</customer>
	<customer>
		<name>Mary</name>
		<age>61</age>
		<country>Ireland</country>
	</customer>
</customers>

This example shows a structure where <customer> represents a record and each record consists of simple XML elements (<name>, <age>, and <country>).

Converting Data

There are many cases where you might need to convert data from flat to hierarchical, or from hierarchical to flat. For example, you may have data flow input in hierarchical format but want the data flow to output flat data. You may also need to convert flat input data to hierarchical data for certain stages (especially stages in Spectrum Spatial) then convert the data back to flat data for output.

To convert data from flat to hierarchical you can use:

  • The Process List tool
  • The Aggregator stage in a flow

To convert data from hierarchical to flat use the Splitter stage.