Uploads files to a server using the S3 protocol.
The S3 nodes enable you to access data from AWS S3 buckets, so that you can integrate your data flows with Amazon. See:
Uploading a single file to Amazon S3
- In the Bucket property, specify the name of the AWS bucket to which you want to upload your file.
- In the FileName property, specify the location of the file to upload to AWS.
Uploading multiple files to Amazon S3
- Drag a Directory List node onto the canvas and connect it to an S3 Put node. Run the Directory List node to generate a list of files in a given location.
- On the S3 Put node, in the Bucket property specify the name of the AWS bucket.
- Select the (from Field) variant of the FileName property, then specify the name of the input field that references the location of the files to upload.
Example
You have four files on your local desktop that you want to upload to Amazon S3.
- Drag a Directory List node onto the canvas and in the DirectoryName property type the location of the folder that contains the four files, e.g.
C:\Users\<username>\Documents\S3 upload. - You want to upload all files in the folder, so in the Pattern property type:
*.* - Run the Directory List node, then connect it to anS3 Put node.
The output of the Directory List node is as follows:
FileName
unicode
File
unicode
Size
long
Created
datetime
Modified
datetime
C:\Users\<username>\Documents\S3 upload\create4.txt create4.txt 25 2017-10-27T06:50:43-0400 2017-10-27T06:50:43-0400 C:\Users\<username>\Documents\S3 upload\create3.txt create3.txt 25 2017-10-27T06:50:43-0400 2017-10-27T06:50:43-0400 C:\Users\<username>\Documents\S3 upload\create2.txt create2.txt 25 2017-10-27T06:50:43-0400 2017-10-27T06:50:43-0400 C:\Users\<username>\Documents\S3 upload\create1.txt create1.txt 25 2017-10-27T06:50:43-0400 2017-10-27T06:50:43-0400 - On the S3 Put node, in the Bucket property, type the name of the AWS bucket that you want to upload your files to, e.g.
my.example.bucket - Select the (from Field) variant of the Object property and type the name of the input field containing the files that you want to upload, in this case
File. - Select the (fromField) variant of the FileName property and type the name of the input field containing the location of the files that you want to upload, in this case
FileName. - Run the S3 Put node to upload your files.
By default, authentication is based on the AWS Default Credential Provider Chain. Alternatively, you can choose to specify an AccessKey and a SecretKey, or for cross-account access (for example when your S3 bucket is located in a different account to your Data360 Analyze instance) you can use instance profiles by configuring the RoleARN and RoleSessionName properties. For more information, see https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/
Properties
Bucket
Specify the name of the AWS Bucket.
Choose the (from Field) variant of this property to look up the value from an input field with the name specified.
A value is required for this property.
Object
Optionally specify the name of the AWS Object.
Choose the (from Field) variant of this property to look up the value from an input field with the name specified.
The default value is the base name of the file.
AccessKey
Optionally specify the AWS Access Key.
If the Access Key is not set then the AWS default strategy will be used.
SecretKey
Optionally specify the AWS Secret Key.
RoleARN
Optionally specify the Amazon Resource Name (ARN) of the role to assume when using cross account access using Instance Profiles.
If the RoleARN (and AccessKey) are not set then authentication will fall back to the AWS Default Credential Provider Chain. For more information, see https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html.
RoleSessionName
Optionally specify the role session name used to uniquely identify a session when the same role is assumed by different principals or for different reasons.
In cross-account scenarios, the role session name is visible to, and can be logged by the account that owns the role.
Region
Optionally specify the AWS region.
If the Region is not set then the AWS default strategy will be used - see AWS documentation on DefaultAWSCredentialsProviderChain.
Encryption
Optionally specify whether to encrypt (server-side) the uploaded file.
The default value is False.
Endpoint
Optionally specify the URL endpoint used to connect to other system services. The default is amazonaws.com endpoint.
The URL should contain the protocol (http or https), the host, and the port.
Depending on Region entered the default value is s3.us-east-2.amazonaws.com for region us-east-2.
FileName
Specify the location of the file to upload to AWS.
Choose the (from Field) variant of this property to look up the value from an input field with the name specified.
A value is required for this property.
ProxyURL
Optionally specify the URL of the proxy server to use for requests. The URL should contain the protocol (http or https), the host, and the port. All other parts of the URL will be ignored.
The URL should be in a format similar to the following:
HTTP proxy: http://proxy.example.com:1234
HTTPS proxy: https://proxy.example.com:443
ProxyUsername
Optionally specify the proxy username, where the proxy requires authentication.
This property should be left blank if ProxyURL is not set.
ProxyPassword
Optionally specify the proxy password, where the proxy requires authentication.
This property should be left blank if ProxyURL is not set.
IgnoreSystemProxy
If proxy settings are not defined then the system proxy is automatically used. This optional property tells Data360 Analyze to ignore the system proxy when set to true.
The default value is False.
FailureBehavior
Optionally specify what to do when a file fails to upload. Choose from:
- Error - Report error and stop further processing.
- Log - Log a warning message and skip the file.
- Ignore - Skip the file.
The default value is Log.
Inputs and outputs
Inputs: 1 optional.
Outputs: uploaded files, errors.