S3 Connector - CData Arc Documentation

The S3 connector integrates with Amazon’s S3 (Simple Storage Service) and other S3-like services (such as Google Storage and Wasabi).

Key Capabilities

Amazon S3 and S3-compatible service integration (Google Storage, Wasabi) with IAM role and access key authentication
Bucket-based file organization with bidirectional transfers and prefix-based virtual folders
Client-side and server-side encryption options with configurable access policies
Optional caching to ensure only new or updated files are downloaded

Overview

Each S3 connector can automatically upload to and download from a single S3 bucket. Before you begin, you need an Amazon account with the appropriate credentials (or account credentials for the S3-like service you are using). Specify the upload and download paths in the bucket. The connector supports download filters by file name.

Connector Configuration

This section contains all of the configurable connector properties.

Settings Tab

Host Configuration

Settings related to the remote connection target.

Setting	Description
Connector Id	The static, unique identifier for the connector.
Connector Type	Displays the connector name and a description of what it does.
Connector Description	An optional field to provide a free-form description of the connector and its role in the flow.
Service	Use the dropdown to choose which service to connect to. Select Other to specify the base URL to use when connecting to the service.
Bucket Name	The S3 bucket to poll or upload to.
Region	The Region where the specified Bucket Name is stored.

Account Settings

Settings related to the account with permission to access the configured Bucket Name.

Setting	Description
IAM Role	Whether to use the attached IAM role to access S3. Only use this setting when CData Arc is hosted on an EC2 instance that has an IAM role attached. The IAM credentials replace the two Key options below.
Access Key	The Access Key account credential acquired from Amazon (or the S3-like service).
Secret Key	The Secret Key account credential acquired from Amazon (or the S3-like service).
Assume Role ARN	Use the two Key options above to call the Amazon STS service to obtain temporary credentials to access S3 with the provided role ARN.

TLS Settings

Settings related to TLS negotiation with the S3 server.

Setting	Description
TLS	Check this to enable TLS negotiation.
Server Public Certificate	The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the server requires a specific certificate for validation. If the server does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server’s identity.

Upload

Settings related to the path in the specified bucket where files are uploaded.

Setting	Description
Prefix	The remote path on the server where files are uploaded.
Overwrite Action	Whether to overwrite, skip, or fail existing files.

Download

Settings related to the path in the specified bucket where files are downloaded.

Setting	Description
Prefix	The remote path on the server from where files are downloaded.
File Filter	A glob pattern filter to determine which files should be downloaded from the remote storage (for example, .txt). You can use negative patterns to indicate files that should not* be downloaded (for example, -*.tmp). Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found.
Delete	Check this to delete successfully downloaded files from the remote storage.

Caching

Settings related to caching and comparing files between multiple downloads.

Setting	Description
File Size Comparison	Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download.
Timestamp Comparison	Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download.

Arc はリソースキャッシュを自動的にクリーンアップします。この処理はすべての受信サイクルの間にバックグラウンドで実行され、リモートサーバー上にすでに存在しないファイルの古いキャッシュエントリを削除します。これにより、手動での操作を必要とせずにキャッシュサイズを管理可能な状態に保ちます。

デフォルトでは、ファイルは30 日間保持されますが、Advanced タブのOther Settings フィールドにあるResourceCacheRetentionDays 設定を使用して、お使いの環境に合わせて調整できます。例えば、ResourceCacheRetentionDays=15 とすると、ファイルは削除されるまで15 日間保持されます。

When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between TEST.TXT and test.txt.

Advanced Tab

Advanced Settings

Settings not included in the previous categories.

Setting	Description
Access Policy	The access policy set on objects after they are uploaded to the S3 server.
Encryption Password	If set, object data is encrypted on the client side before upload, and downloaded objects are automatically decrypted.
Recurse	Whether to download files in subfolders of the target remote path.
Local File Scheme	A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see Macros.
Server Side Encryption	Whether to use server-side AES256 encryption.
TLS Enabled Protocols	The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2.
Virtual Hosting	Whether to use hosted-style or path-style requests when referencing the bucket endpoint.
Processing Delay	The amount of time (in seconds) by which the processing of files placed in the Transactions tab is delayed. This is a legacy setting. Best practice is to use a File connector to manage local file systems instead of this setting.

Proxy Settings

これらは、接続のルーティング先となるプロキシを識別し認証するための一連の設定です。デフォルトでは、このセクションはSecurity Settings ページのProxy Settings 部分にあるグローバル設定を使用します。チェックボックスをオフにすると、コネクタ固有の設定を指定できます。

設定	説明
Proxy Type	プロキシベースのファイアウォールで使用されるプロトコル。
Proxy Host	プロキシベースのファイアウォールの名前またはIP アドレス。
Proxy Port	プロキシベースのファイアウォールのTCP ポート。
Proxy User	プロキシベースのファイアウォールで認証するために使用するユーザー名。
Proxy Password	プロキシベースのファイアウォールで認証するために使用するパスワード。
Authentication Scheme	デフォルトのNone のままにするか、次の認証スキームのいずれかを選択します：Basic、Digest、Proprietary、またはNTLM。

Message

Message 設定は、コネクタがメッセージを検索する方法と、処理後にメッセージを管理する方法を決定します。

Note： 以下の設定は非推奨となり、デフォルトで非表示になっています。以前に有効化またはデフォルト以外の値に設定されていたコネクタにのみ表示されます。処理済みファイルのコピーを保持するには、フロー内のコネクタを右クリックし、成功パスを表示を選択して、成功パスをFile コネクタに接続してください。

設定	説明
Sent フォルダに保存（非推奨）	コネクタが処理したファイルをSent フォルダにコピーします。デフォルトで無効です。Sent フォルダに保存されたファイルは保存データの暗号化の対象外です。
Sent フォルダのスキーム（非推奨）	選択した時間間隔でSent フォルダ内のファイルをグループ化します。Sent フォルダに保存が有効な場合のみ適用されます。

Logging

ログの作成と保存を制御する設定です。

設定	説明
Log Level	コネクタが生成するログの詳細度です。サポートを依頼する際は、これをDebug に設定してください。
Log Subfolder Scheme	選択した間隔に従って、コネクタがLogs フォルダ内のファイルをグループ化するよう指示します。Weekly オプション（デフォルト）は、コネクタが毎週新しいサブフォルダを作成し、その週のすべてのログをそのフォルダに保存するよう指示します。この設定を空白のままにすると、コネクタはすべてのログをLogs フォルダに直接保存します。多数のトランザクションを処理するコネクタでは、サブフォルダを使用することでログを整理しやすくなり、パフォーマンスが向上します。
Log Messages	処理したファイルのログエントリにファイル自体のコピーを含めるには、これをオンにします。これを無効にすると、Transactions タブからファイルのコピーをダウンロードできなくなる場合があります。

Miscellaneous

Miscellaneous 設定は、特定のユースケース向けの設定です。

設定	説明
Other Settings	非表示のコネクタ設定をセミコロン区切りのリストで設定できます（例：`setting1=value1;setting2=value2`）。通常のコネクタのユースケースや機能では、これらの設定の使用は必要ありません。

Automation Tab

Automation Settings

Settings related to the automatic processing of files by the connector.

Setting	Description
Send	Whether files arriving at the connector are automatically uploaded.
Retry Interval	The amount of time before a failed upload is retried.
Max Attempts	The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connector retries the file indefinitely.
Receive	Whether the connector should automatically poll the remote download path for files to download.
Interval	The interval between automatic download attempts.
Minutes Past the Hour	The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to Hourly. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc.
Time	The time of day that the attempt should occur. Only applicable when the interval setting above is set to Daily, Weekly, or Monthly.
Day	The day on which the attempt should occur. Only applicable when the interval setting above is set to Weekly or Monthly.
Minutes	The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to Minute.
Cron Expression	A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to Advanced.
Start Date and Time	The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the Automation Status section.

Automation Status

Performance

コネクタへのリソースの割り当てに関する設定です。

設定	説明
Max Workers	このコネクタでファイルを処理するためにスレッドプールから消費されるワーカースレッドの最大数です。設定すると、Advanced Settings ページのPerformance Settings 部分にあるデフォルト設定が上書きされます。
Max Files	コネクタに割り当てられた各スレッドが送信するファイルの最大数です。設定すると、Advanced Settings ページのPerformance Settings 部分にあるデフォルト設定が上書きされます。

Alerts Tab

SLAs Tab

Establishing a Connection

The requirements for establishing an S3 connection are simple:

Amazon account credentials (or other S3-like account credentials)
- Access Key
- Secret Key
A bucket that can be accessed by the above account

For Amazon S3, use this link to obtain Access Key and Secret Key information from Amazon. Optionally, you can secure the connection with S3 servers with TLS by enabling the Use TLS option in the TLS Settings section.

Uploading

Upload to Remote Folders

The Prefix setting in the Upload section of the Settings page specifies the bucket path to upload files to. This allows for the logical separation of files into virtual folders in the same bucket.

S3 servers do not maintain a real folder structure, and Arc uses application logic to present a pseudo folder structure. Slashes in the Prefix (/, \\) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from ‘subfolders’ in the bucket based on the slashes in the path.

Upload Automation

The S3 connector supports automatic upload via the Automation tab. When Upload automation is enabled, files that reach the Transactions tab for the connector are automatically uploaded to the specified Bucket Name at the specified Prefix. If a file fails to upload, the application attempts to send it again after the Retry Interval has elapsed. This process continues until the Max Attempts has been reached, after which the connector raises an error.

Downloading

Download from Remote Folders

The Prefix setting in the Download section of the Settings page specifies the bucket path to download files from. This allows for the logical separation of files into virtual folders in the same bucket. The File Filter setting provides a way to only download specific filenames in the specified path.

Download Automation

The S3 connector supports automatic download via the Automation tab. When Download automation is enabled, the connector automatically polls the remote bucket based on the specified Download Interval.

​Key Capabilities

​Overview

​Connector Configuration

​Settings Tab

​Host Configuration

​Account Settings

​TLS Settings

​Upload

​Download

​Caching

​Advanced Tab

​Advanced Settings

​Proxy Settings

​Message

​Logging

​Miscellaneous

​Automation Tab

​Automation Settings

​Automation Status

​Performance

​Alerts Tab

​SLAs Tab

​Establishing a Connection

​Uploading

​Upload to Remote Folders

​Upload Automation

​Downloading

​Download from Remote Folders

​Download Automation

​Macros

​Examples

Key Capabilities

Overview

Connector Configuration

Settings Tab

Host Configuration

Account Settings

TLS Settings

Upload

Download

Caching

Advanced Tab

Advanced Settings

Proxy Settings

Message

Logging

Miscellaneous

Automation Tab

Automation Settings

Automation Status

Performance

Alerts Tab

SLAs Tab

Establishing a Connection

Uploading

Upload to Remote Folders

Upload Automation

Downloading

Download from Remote Folders

Download Automation

Macros

Examples