FTP Put - Data360_Analyze - Latest

Data360 Analyze Server Help
Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 Analyze
Version
Latest
Language
English
Product name
Data360 Analyze
Title
Data360 Analyze Server Help
Copyright
2024
First publish date
2016
Last updated
2024-11-28
Published on
2024-11-28T15:26:57.181000

Uploads files to a server using the FTP, FTPS, or SFTP protocol.

If you have a list of files to upload, you can pass it to the node on an input pin. Alternately, a single file or directory to upload can be specified within the node.

Filenames on the FTP server are never relative to the FTP user's home directory; they must be absolute paths (begin with a leading slash "/"), where the root directory is the top level directory the FTP user can access.

Server paths (i.e. paths given in the ServerPath property) cannot contain backslashes; forward slashes must be used as the file separator.

Example data (assume these files for the examples below):


/LocalDir/Dir1/File1
/LocalDir/Dir1/File2
/LocalDir/Dir1/Dir2/File3
/LocalDir/Dir1/Dir2/File4
/LocalDir/Dir1/Dir3/File5

Example 1: Uploading a File

We want to upload the file "/LocalDir/Dir1/File1" to "/ServerDir/".


LocalPath:   /LocalDir/Dir1/File1
ServerPath:  /ServerDir/

The file is created as:


/ServerDir/File1

Example 2: Uploading a Directory

We want to upload the directory "/LocalDir/Dir1" to "/ServerDir/".


LocalPath:   /LocalDir/Dir1
Pattern:     *
ServerPath:  /ServerDir/

The following files are created: 


/ServerDir/Dir1/File1
/ServerDir/Dir1/File2

Note that "Dir2" and "Dir3" were not uploaded. If we want to include them, we can set Recurse="true".

Example 3: Uploading a Directory's Contents

We want to upload the contents of "/LocalDir/Dir1" directly to "/ServerDir".


BaseLocalDir:  /LocalDir/Dir1
LocalPath:     /LocalDir/Dir1 (or LocalPath=.)
Pattern:       *
ServerPath:    /ServerDir

The following files are created: 


/ServerDir/File1
/ServerDir/File2

Example 4: Uploading Part of a Directory Tree

We only want to upload a few files from within a directory tree, but we want to retain the directory structure.


BaseLocalDir:  /LocalDir
ServerPath:    /ServerDirLocalPath: (read from input pin)
/LocalDir/Dir1/File1
/LocalDir/Dir1/File2
/LocalDir/Dir1/Dir3/File5
</example.The following files are created:
<example>/ServerDir/Dir1/File1
/ServerDir/Dir1/File2
/ServerDir/Dir1/Dir3/File5

Note the LocalPath could also be given as the following. with the same result: 


./Dir1/File1
./Dir1/File2
./Dir1/Dir3/File5

We can get the same result without using the BaseLocalDir property (to do so, we need to specify a ServerPath for each file, e.g. "/ServerDir/Dir1/File1" for File1, and use absolute paths for LocalPath). Using BaseLocalDir saves us the work of needing to build a ServerPath for each file.

Properties

Server

Specify the name of server to connect to (e.g. ftp-server.example.com).

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

ServerPort

Optionally specify the port the server is running on.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is 21 for FTP and FTPS; 22 for SFTP.

ServerUsername

Optionally specify the username to login with.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerPassword

Optionally specify the password to login with.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

LocalPath

Specify the path of file or directory to upload. This must be an absolute path (unless a BaseLocalDir is specified).

When uploading a directory, a value for the Pattern property must be specified as well.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

ServerPath

Specify the path of server file to create, or server directory to upload into. This must be an absolute path

Directories must be indicated with a trailing slash, e.g. /DstDir/

You can rename the file by giving the new name here, e.g. /DstDir/NewFileName

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

A value is required for this property.

Protocol

Optionally specify the protocol used to connect to the remote server. Choices are:

  • FTP - Not encrypted.
  • FTPS - Explicit FTPS; this is an FTP session which is encrypted using the "AUTH TLS" command.
  • SFTP - The "SSH File Transfer Protocol", an encrypted protocol.

The default value is FTP.

BaseLocalDir

Optionally specify an absolute directory path. When BaseLocalDir is set, files are copied to "<ServerPath>/<Relative path to local file from BaseLocalDir>". This is useful if we are selectively copying some files from a tree and want to preserve the directory structure.

The LocalPath can be given either as an absolute or relative path (relative to BaseLocalDir).

See examples of how to use this property in the node help:

  • Uploading a Directory's Contents
  • Uploading Part of a Directory Tree

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

Recurse

Optionally specify whether to search the directory tree under LocalPath for files that match the Pattern property.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is False.

RecursionDepth

Optionally specify a limit on how many directory levels to recurse into.

For example, assume the directory "/Dir1/Dir2/Dir3". If LocalPath="/Dir1", Pattern=*, Recurse="true":

  • RecursionDepth 1 searches only the files contained in /Dir1, i.e. "/Dir1/*"
  • RecursionDepth 2 is "Dir1/*/*", i.e. searches /Dir1/* and /Dir1/Dir2/*
  • RecursionDepth 3 is "Dir1/*/*/*"

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is infinity; ie. recurse as deeply as possible.

Pattern

Optionally specify an expression to select which files to upload.

For example, if we have a directory full of files named Data1, Data2, Data3, ...; we can use "Data*" to match all of them. See the PatternType property for more details.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

PatternMode

Optionally specify how to use the Pattern property to exclude files from being uploaded. Choose from:

  • Include- Upload files that match the Pattern.
  • Exclude- Upload files that do not match the Pattern.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is Include.

PatternType

Optionally specify the syntax to use in the Pattern property. Choose from:

  • Wildcard - Similar to the syntax used by DOS and BASH globbing. '*' matches any sequence of 0 or more characters; '?' matches a single character.
  • Regex - A regular expression. See the API documentation for java.util.regex.Pattern for details.

For example, using "Wildcard" syntax, "data.???" will match: data.doc, data.txt, etc (but not data.text). "data*" will match data, data.doc, data.txt, data.text, etc.

The regex equivalent of the above examples are: "data\...." and "data.*".

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is Wildcard.

PatternSubject

Optionally specify the part of the file path compared against the Pattern property. Choose from:

  • Filename
  • Path

Path is meaningful only when Recurse is set. When selected, the Pattern is compared to the portion of the path under the LocalPath directory.

Let's say we're trying to upload all files under a directory with "Data" in its name. Assume LocalPath="/LocalDir/", Recurse="true", Pattern="*Data*", and the following files exist: "/LocalDir/Data1/File1" and "/LocalDir/Data2/File5"

Filename means compare the Pattern to the strings "File1" and "File5" (no match)

Path means compare the Pattern to the string "Data1/File1" and "Data2/File5" (matches both)

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

The default value is Filename.

CaseSensitive

Optionally specify whether the pattern and filename case must match, when the Pattern property is used.

Assume a file "/LocalDir/FILE1":

  • If Pattern="file*" and CaseSensitive="false", the file will be uploaded
  • If Pattern="file*" and CaseSensitive="true", the file will not be uploaded
  • If PatternSubject="Path", then the CaseSensitive property is applied to the portion of the path under the LocalPath directory (see "PatternSubject" for an example).

Otherwise, when not using pattern matching, and also for the part of the path that isn't compared to the pattern, case sensitivity is left up to the local machine. For example, Windows accesses filenames case insensitively.

The default value is False.

CreateDirectories

Optionally specify under what circumstances the node should create directories.

Usually, we transfer files into an existing directory. If the directory does not exist, we want the node to error (e.g. maybe we entered an incorrect destination path). But, sometimes we know the directory does not exist, and we'd like to have it created for us. Choose from:

  • All - Create any needed directories that do not exist. This includes subdirectories that result from using the Recurse property.
  • Subdirectories - Create any needed directories under the destination directory (i.e. LocalPath). Error if the destination directory does not exist.
  • None - Never create a directory. Error whenever a file is being transferred and the directory it's going to does not exist. This option is useful if we are copying a directory tree (e.g. with Recurse), and want to be sure that the directory structure matches exactly.

The default value is Subdirectories as it guards against accidental misplacement of files.

For example, say "/dir1/" is being transferred to "/LocalDir/", with the following property values: ServerPath="/dir1/", Pattern="*", LocalPath="/LocalDir/", and CreateDirectories="Subdirectories". In this case, because the destination "/LocalDir/" exists, the subdirectory "/LocalDir/dir1/" is created. If "/LocalDir/" did not exist, the request would error.

PassThroughFields

Optionally specify which input fields to retain on the output pin. Choose from:

  • All - All input fields are copied to the output pin.
  • Used - Only fields that are referenced in the node.
  • Unused - Only fields that are not referenced in the node.
  • None - No input fields are copied to the output pin.

The default value is Used.

EncryptionAlgorithm

Optionally select one of the provided encryption types, or type an encryption type if the one you require is not in the list.

  • Best Available
  • aes128-ctr
  • aes128-cbc
  • 3des-ctr
  • 3des-cbc
  • blowfish-cbc
  • aes192-ctr
  • aes256-ctr
  • aes128-gcm@openssh.com
  • aes256-gcm@openssh.com

Best Available uses the strongest encryption type that both the client and server support. When you specify a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

MacAlgorithm

Optionally select one of the provided MAC (Message Authentication Code) algorithm types, or type a MAC type if the one you require is not in the list.

  • Best Available
  • hmac-md5
  • hmac-sha1
  • hmac-sha2-256
  • hmac-sha1-96
  • hmac-md5-96
  • hmac-sha2-512
  • hmac-sha2-512-etm@openssh.com

Best Available uses the strongest encryption type that both the client and server support. When you specify a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP, all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

KeyExchangeAlgorithm

Optionally select one of the provided Key Exchange algorithm types, or type a Key Exchange type if the one you require is not in the list.

  • Best Available
  • curve25519-sha256
  • curve25519-sha256@libssh.org
  • ecdh-sha2-nistp256
  • ecdh-sha2-nistp384
  • ecdh-sha2-nistp521
  • diffie-hellman-group16-sha512
  • diffie-hellman-group18-sha512

Best Available uses the strongest algorithm type that the server supports. When you select a specific algorithm, the node will fail if the server does not support the algorithm.

Valid options for this property vary depending on the protocol selected.

For protocol SFTP, all options are valid.

For protocol FTPS, Best Available is the only valid choice (FTPS does not have an option to choose an algorithm).

For protocol FTP, selecting anything other than "(default)" will cause an error (plain FTP is not encrypted).

The default value is Best Available.

PassiveMode

Optionally specify the FTP transfer mode to be used. When set to True the node executes in Passive FTP mode When set to False the node executes in Active FTP mode.

The default value is False i.e. Active FTP mode.

ConnectionTimeout

Optionally specify the maximum period (in seconds) to wait before the connection times out.

By default, a connection timeout period does not apply.

ErrorThreshold

Optionally specify the number of transfer errors that will cause the node to give up and fail.

Each record on the input pin is a "request". A transfer error is any error that causes a request to fail (e.g. a requested file does not exist). Setting this property instructs the node to continue processing requests as long as the number of errors remains below the given threshold.

A value of "0" means "never fail on a transfer error" (the node will still fail on more serious errors).

The default value is "1" - the node fails on the first error encountered.

FileExistsBehavior

Optionally specify what to do when a file being downloaded already exists on the local machine. Choose from:

  • Error - Give a transfer error and skip the file.
  • Log - Log a warning message and skip the file.
  • Ignore - Skip the file.
  • Overwrite - Overwrite the file.
  • Update - Overwrite if the file being downloaded is newer than the existing file.

The default value is Error.

ErrorPassThroughFields

Optionally specify the input fields to retain on the error output pin. Choose from:

  • All - All input fields are copied to the error pin.
  • Used - Only fields that are referenced in the node.
  • None - No input fields are copied to the error pin.

The default value is None.

StoreBatchSize

Optionally specify the maximum number of requests per batch.

Each record on the input pin is a "request". This property determines how many requests are grouped together and processed at once.

Note that a batch only contains requests to the same server. So, batches may contain fewer requests than StoreBatchSize. For example, if StoreBatchSize is 10, and we have requests to the following Servers: A, A, B, A; then the requests will be processed in 3 batches: (A, A), (B), and (A).

A value of 0 means "infinity" - ie. group together as many requests as possible.

The default value is 1, meaning each request is processed in its own batch.

StoreCacheSize

Optionally specify the number of server connections to keep open.

For example, if set to 5, then the 5 most recent server connections are kept open. When a request is made to a server in the cache, the node will re-use the open connection, avoiding the overhead of opening a new connection.

If set to 0, then no connections are cached. That is, the server connection is opened at the beginning of a batch and closed at the end of the batch.

The default value is 1.

Inputs and outputs

Inputs: 1 optional.

Outputs: uploaded files, errors.