FTP Get - Data360_Analyze - Latest

Data360 Analyze Server Help
Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 Analyze
Version
Latest
ft:locale
en-US
Product name
Data360 Analyze
ft:title
Data360 Analyze Server Help
Copyright
2025
First publish date
2016
ft:lastEdition
2025-02-20
ft:lastPublication
2025-02-20T11:13:02.494000

Downloads files from a server using the FTP, FTPS, or SFTP protocol.

If you have a list of files to download, you can pass it to the node on an input pin. Alternatively, a single file or directory to download can be specified within the node.

Filenames on the FTP server are never relative to the FTP user's home directory; they must be absolute paths (begin with a leading slash "/"), where the root directory is the top level directory the FTP user can access.

Server paths (i.e. paths given in the ServerPath and BaseServerDir properties) cannot contain backslashes; forward slashes must be used as the file separator.

Example data (assume these files for the examples below): 


/ServerDir/Dir1/File1
/ServerDir/Dir1/File2
/ServerDir/Dir1/Dir2/File3
/ServerDir/Dir1/Dir2/File4
/ServerDir/Dir1/Dir3/File5

Example 1: Downloading a File

We want to download the file "/ServerDir/Dir1/File1" to "/LocalDir/".


ServerPath:  /ServerDir/Dir1/File1
LocalPath:   /LocalDir/

The file is created as: 


/LocalDir/File1

Example 2: Downloading a Directory

We want to download the directory "/ServerDir/Dir1" to "/LocalDir/".

ServerPath:  /ServerDir/Dir1
Pattern:     *
LocalPath:   /LocalDir/

The following files are created: 


/LocalDir/Dir1/File1
/LocalDir/Dir1/File2

Note that "Dir2" and "Dir3" were not downloaded. If we want to include them, we can set Recurse="true".

Example 3: Downloading a Directory's Contents

We want to download the contents of "/ServerDir/Dir1" directly to "/LocalDir".

BaseServerDir:  /ServerDir/Dir1
ServerPath:         /ServerDir/Dir1 (or ServerPath=.)
Pattern:            *
LocalPath:          /LocalDir

The following files are created: 


/LocalDir/File1
/LocalDir/File2

Example 4: Downloading Part of a Directory Tree

We only want to download a few files from within a directory tree, but we want to retain the directory structure.


BaseServerDir:  /ServerDir
LocalPath:          /LocalDirServerPath: (read from input pin)
/ServerDir/Dir1/File1
/ServerDir/Dir1/File2
/ServerDir/Dir1/Dir3/File5

The following files are created: 


/LocalDir/Dir1/File1
/LocalDir/Dir1/File2
/LocalDir/Dir1/Dir3/File5

Note the ServerPath could also be given as the following, with the same result: 


./Dir1/File1
./Dir1/File2
./Dir1/Dir3/File5

We can get the same result without using the BaseServerDir property (to do so, we need to specify a LocalPath for each file, e.g. "/LocalDir/Dir1/File1" for File1, and use absolute paths for "ServerPath"). Using BaseServerDir saves us the work of needing to build a LocalPath for each file.

Properties

Server

Specify the name of server to connect to (e.g. ftp-server.example.com).

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

ServerPort

Optionally specify the port number the server is running on.

The default value is 21 for FTP and FTPS; 22 for SFTP.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerUsername

Specify the username to login with.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerPassword

Specify the password to login with.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerPath

Specify the path of the file or directory to download. Must be an absolute path (unless BaseServerDir is specified). When downloading a directory, a value for the Pattern property must be specified as well.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

LocalPath

Specify the path of a local file to create, or local directory to download into. Must be an absolute path. Directories must be indicated with a trailing slash, e.g. /DstDir/

You can rename the file by giving the new name here, e.g. /DstDir/NewFileName.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

Protocol

Optionally specify the protocol used to connect to the remote server. Choose from:

  • FTP - Not encrypted
  • FTPS - Explicit FTPS; this is an FTP session which is encrypted using the "AUTH TLS" command
  • SFTP - SSH File Transfer Protocol: an encrypted protocol

The default value is FTP.

BaseServerDir

Optionally specify an absolute directory path.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

When BaseServerDir is set, files are copied to "<LocalPath>/<Relative path to server file from BaseServerDir>". This is useful if we are selectively copying some files from a tree and want to preserve the directory structure.

The ServerPath can be given either as an absolute or relative path (relative to BaseServerDir).

Recurse

Optionally specify whether to search the directory tree under ServerPath for files that match the Pattern property.

The default value is False.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

RecursionDepth

Optionally specify a limit on how many directory levels to recurse into.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

For example, assume the directory "/Dir1/Dir2/Dir3". If ServerPath="/Dir1", Pattern=*, Recurse="true":

RecursionDepth 1 searches only the files contained in /Dir1, i.e. "/Dir1/*"

RecursionDepth 2 is "Dir1/*/*", i.e. searches /Dir1/* and /Dir1/Dir2/*

RecursionDepth 3 is "Dir1/*/*/*"

The default value is infinity; ie. recurse as deeply as possible.

Pattern

Optionally specify an expression to select which files to download.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

For example, if we have a directory full of files named Data1, Data2, Data3, ...; we can use "Data*" to match all of them. See the PatternType property for more details.

PatternMode

Optionally specify whether to use the Pattern property to exclude files from being downloaded. Choose from:

  • Include- Download files that match the pattern.
  • Exclude - Download files that do not match the pattern.

The default value is Include.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

PatternType

Optionally specify the syntax used in the Pattern property. Choose from:

  • Wildcard - Similar to the syntax used by DOS and BASH globbing. '*' matches any sequence of 0 or more characters; '?' matches a single character.
  • Regex - Regular expression. See the API documentation for java.util.regex.Pattern for details.

The default value is Wildcard.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

For example, using Wildcard syntax, "data.???" will match: data.doc, data.txt, etc (but not data.text). "data*" will match data, data.doc, data.txt, data.text, etc. The regex equivalent of the above examples are: "data\...." and "data.*".

PatternSubject

Optionally specify the part of the file path compared against the Pattern property. Choose from:

  • Filename - Compare the pattern to the strings "File1" and "File5" (no match)
  • Path - Compare the pattern to the string "Data1/File1" and "Data2/File5" (matches both)

The default value is Filename.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

Path is meaningful only when Recurse is set. When selected, the Pattern is compared to the portion of the path under the ServerPath directory.

Let's say we're trying to download all files under a directory with "Data" in its name. Assume ServerPath="/ServerDir/", Recurse="true", Pattern="*Data*", and the following files exist: "/ServerDir/Data1/File1" and "/ServerDir/Data2/File5"

CaseSensitive

Optionally specify whether the pattern and filename's case must match, when using the Pattern property.

Assume a file "/ServerDir/FILE1":

If Pattern="file*" and CaseSensitive="false", the file will be downloaded

If Pattern="file*" and CaseSensitive="true", the file will not be downloaded

If PatternSubject="Path", then the CaseSensitive property is applied to the portion of the path under the ServerPath directory (see "PatternSubject" for an example).

Otherwise, when not using pattern matching, and also for the part of the path that isn't compared to the pattern, case sensitivity is left up to the FTP server. For example, a Windows FTP server will respond case insensitively to a request for a specific filename.

The default value is False.

CreateDirectories

Optionally specify under what circumstances the node should create directories.

Usually, we transfer files into an existing directory. If the directory does not exist, we want the node to error (e.g. maybe we entered an incorrect destination path). But, sometimes we know the directory does not exist, and we'd like to have it created for us. Choose from:

  • All - Create any needed directories that do not exist. This includes subdirectories that result from using the Recurse property.
  • Subdirectories - Create any needed directories under the destination directory (i.e. LocalPath). Error if the destination directory does not exist.
  • None - Never create a directory. Error whenever a file is being transferred and the directory it's going to does not exist. This option is useful if we are copying a directory tree (e.g. with Recurse), and want to be sure that the directory structure matches exactly.

The default value is Subdirectories as it guards against accidental misplacement of files.

For example, say "/dir1/" is being transferred to "/LocalDir/", with the following property values: ServerPath="/dir1/", Pattern="*", LocalPath="/LocalDir/", and CreateDirectories="Subdirectories". In this case, because the destination "/LocalDir/" exists, the subdirectory "/LocalDir/dir1/" is created. If "/LocalDir/" did not exist, the request would error.

PassThroughFields

Optionally specify the input fields to retain on the output pin. Choose from:

  • All - All input fields are copied to the output pin.
  • Used - Only fields that are referenced in the node.
  • Unused - Only fields that are not referenced in the node.
  • None - No input fields are copied to the output pin.

The default value is Used.

EncryptionAlgorithm

Optionally select one of the provided encryption types, or type an encryption type if the one you require is not in the list.

  • Best Available
  • aes128-ctr
  • aes128-cbc
  • 3des-ctr
  • 3des-cbc
  • blowfish-cbc
  • aes192-ctr
  • aes256-ctr
  • aes128-gcm@openssh.com
  • aes256-gcm@openssh.com

Best Available uses the strongest encryption type that both the client and server support. When you specify a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

MacAlgorithm

Optionally select one of the provided MAC (Message Authentication Code) algorithm types, or type a MAC type if the one you require is not in the list.

  • Best Available
  • hmac-md5
  • hmac-sha1
  • hmac-sha2-256
  • hmac-sha1-96
  • hmac-md5-96
  • hmac-sha2-512
  • hmac-sha2-512-etm@openssh.com

Best Available uses the strongest encryption type that both the client and server support. When you specify a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP, all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

KeyExchangeAlgorithm

Optionally select one of the provided Key Exchange algorithm types, or type a Key Exchange type if the one you require is not in the list.

  • Best Available
  • curve25519-sha256
  • curve25519-sha256@libssh.org
  • ecdh-sha2-nistp256
  • ecdh-sha2-nistp384
  • ecdh-sha2-nistp521
  • diffie-hellman-group16-sha512
  • diffie-hellman-group18-sha512

Best Available uses the strongest algorithm type that the server supports. When you select a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP, all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

PassiveMode

Optionally specify the FTP transfer mode to be used. When set to True the node executes in Passive FTP mode When set to False the node executes in Active FTP mode.

The default value is False ie. Active FTP mode.

ConnectionTimeout

Optionally specify the maximum period (in seconds) to wait before the connection times out.

By default, a connection timeout period does not apply.

FileExistsBehavior

Optionally specify what to do when a file being downloaded already exists on the local machine. Choose from:

  • Error - Give a transfer error and skip the file.
  • Log - Log a warning message and skip the file.
  • Ignore - Skip the file.
  • Overwrite - Overwrite the file.
  • Update - Overwrite if the file being downloaded is newer than the existing file.

The default value is Error.

ErrorThreshold

Optionally specify the number of transfer errors that will cause the node to give up and fail.

Each record on the input pin is a "request". A transfer error is any error that causes a request to fail (e.g. a requested file does not exist). Setting this property instructs the node to continue processing requests as long as the number of errors remains below the given threshold.

A value of "0" means "never fail on a transfer error" (the node will still fail on more serious errors).

The default value is "1" - the node fails on the first error encountered.

ErrorPassThroughFields

Optionally specify the input fields to retain on the error output pin. Choose from:

  • All - All input fields are copied to the error pin.
  • Used - Only fields that are referenced in the node.
  • None - No input fields are copied to the error pin.

The default value is None.

StoreBatchSize

Optionally specify the maximum number of requests per batch.

Each record on the input pin is a "request". This property determines how many requests are grouped together and processed at once.

Note that a batch only contains requests to the same server. So, batches may contain fewer requests than StoreBatchSize. For example, if StoreBatchSize is 10, and we have requests to the following Servers: A, A, B, A; then the requests will be processed in 3 batches: (A, A), (B), and (A).

A value of 0 means "infinity" - ie. group together as many requests as possible.

The default value is 1, meaning each request is processed in its own batch.

StoreCacheSize

Optionally specify the number of server connections to keep open.

For example, if set to 5, then the 5 most recent server connections are kept open. When a request is made to a server in the cache, the node will re-use the open connection, avoiding the overhead of opening a new connection.

If set to 0, then no connections are cached. That is, the server connection is opened at the beginning of a batch and closed at the end of the batch.

The default value is 1.

Inputs and outputs

Inputs: 1 optional (input fields).

Outputs: downloaded files, errors.