Directory List - Data360_Analyze - 3 - 3.12

Data360 Analyze Server Help

Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 Analyze
Version
3.12
Language
English
Product name
Data360 Analyze
Title
Data360 Analyze Server Help
Copyright
2023
First publish date
2016

Scans the specified directories and outputs a list of files that meet the specified criteria.

Often used in conjunction with a second acquisition node, the Directory List node is the first step in importing multiple files of the same type.

This saves you having to configure a separate data acquisition node for each file that you want to import, and instead allows you to combine data from multiple sources into one data source which is then processed by a second data acquisition (Input Connector) node.

Importing files from a single directory

  1. In the DirectoryName property, enter the location of the directory where the files that you want to import are located, for example: C:/Users/UserName/acquisitionData
  2. In the Pattern property, enter a file name and the file type that is common to all files that you want to import. You can use standard wildcards to import all files that follow a certain pattern, for example:*.xlsxThe * tells the node to include all .xlsx files, regardless of the file name.

Importing files from multiple directories

In this example, you have a list of directories in a Create Data node. You want to import all .CSV files in the listed directories.

The Create Data node contains the following in the Data property:

Files:stringC:/Users/<username>/Documents/Directory List exampleC:/Users/<username>/Documents/Directory List example/SubdirectoryC:/Users/<username>/Different directory

  1. Run the Create Data node and connect it to the input pin of the Directory List node.
  2. Select the (from Field) variant of the DirectoryName property.
  3. Click the menu button in the DirectoryName property and from the Input Fields menu, select the field that contains the input data, in this case Files.
  4. As you want to import all CSV files in the listed directories, in the Pattern property type *.csv.
Tip: If there are any subdirectories or specific files that you want to exclude, you can specify these in the ExcludeSubdirectories and ExcludeFilePatterns properties.

The Directory List node outputs a list of filenames of the specified type in the specified directories, along with the following additional information (there may also be additional fields depending on the value set in the PassThroughFields property):

  • File created/modified times
  • File size (bytes)
  • Full file paths

After running the Directory List node, you then need to add a second acquisition node to import the data, for example, the Excel File node or the CSV/Delimited Input node.

Properties

DirectoryName

Specify the directory where the files that you want to import are located. For example: C:\Users\UserName\acquisitionData.

Choose the (from Field) variant of this property to specify the name of an input field containing one or more directories where the files that you want to import are located.

A value is required for this property.

Pattern

Specify the filename and file type pattern which the directory entries will be matched against, for example: *.xlsx. Any files that match the specified criteria will be listed in the output.

A value is required for this property.

CaseInsensitive

Optionally specify whether to use case-insensitive pattern matching (Unix only).

The default value is False.

Recurse

Optionally specify whether to include matches in sub-directories.

The default value is False.

ExcludeSubdirectories

Optionally specify a list of patterns matching subdirectories which are to be excluded from the search.

Each subdirectory pattern to exclude must be entered on a separate line.

This property only has an effect when the Recurse property is set to True.

By default, no directories are excluded.

ExcludeFilePatterns

Optionally specify a list of patterns matching filenames which are to be excluded from the search.

Each filename pattern to exclude must be entered on a separate line.

Files will only be reported by the Directory List node which match the Pattern property and do not match any of the ExcludeFilePatterns.

By default, no files are excluded.

PassThroughFields

Optionally specify which input fields will "pass through" the node unchanged from the input to the output, assuming that the input exists. The input fields specified will appear on those output records which were produced as a result of the input fields. Choose from:

  • All – Passes through all the input data fields to the output.
  • None – Passes none of the input data fields to the output; as such, only the fields created by the node appear on the output.
  • Used – Passes through all the fields that the node used to create the output. Used fields in this case will be a field referenced in the DirectoryName property when using the from Field option.
  • Unused – Passes through all the fields that the node did not use to create the output.

The default value is None.

If a naming conflict exists between a pass through field and an explicitly named output field, an error will occur.

Inputs and outputs

Inputs: 1 optional (in1).

Outputs: Directory List.