> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arc.cdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# S3 Connector

> Configuration and usage guide for the CData Arc S3 connector.

export const CommonCacheCleanup = () => <>
    <p>Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action.</p>
    <p>By default, files are retained for 30 days, but you can use the <code>ResourceCacheRetentionDays</code> setting in the <strong>Other Settings</strong> field on the <strong>Advanced</strong> tab to adjust it to fit your environment. For example, <code>ResourceCacheRetentionDays=15</code> retains files for 15 days before removing them.</p>
  </>;

export const CommonProxySettings = () => <>
    <p>These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the <a href="/26.3/cloud/en/getting-started/administration/settings/proxy-settings">Proxy Settings</a> portion of the <a href="/26.3/cloud/en/getting-started/administration/settings/security">Security Settings</a> page. Clear the checkbox to supply settings specific to your connector.</p>
    <table>
      <thead>
        <tr><th>Setting</th><th>Description</th></tr>
      </thead>
      <tbody>
        <tr>
          <td><strong>Proxy Type</strong></td>
          <td>The protocol used by a proxy-based firewall.</td>
        </tr>
        <tr>
          <td><strong>Proxy Host</strong></td>
          <td>The name or IP address of a proxy-based firewall.</td>
        </tr>
        <tr>
          <td><strong>Proxy Port</strong></td>
          <td>The TCP port for a proxy-based firewall.</td>
        </tr>
        <tr>
          <td><strong>Proxy User</strong></td>
          <td>The user name to use to authenticate with a proxy-based firewall.</td>
        </tr>
        <tr>
          <td><strong>Proxy Password</strong></td>
          <td>A password used to authenticate to a proxy-based firewall.</td>
        </tr>
        <tr>
          <td><strong>Authentication Scheme</strong></td>
          <td>Leave the default <strong>None</strong> or choose from one of the following authentication schemes: <strong>Basic</strong>, <strong>Digest</strong>, <strong>Proprietary</strong>, or <strong>NTLM</strong>.</td>
        </tr>
      </tbody>
    </table>
  </>;

export const SlasTab = ({siteName = "CData Arc"}) => <>
    <p><em>Settings related to configuring Service Level Agreements (SLAs).</em></p>
    <p>
      SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. {siteName} sends emails to warn the user when an SLA is not met, and marks the SLA as <em>At Risk</em>, which means that if the SLA is not met soon, it will be marked as <em>Violated</em>. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again.
    </p>
    <p>
      To define an SLA, toggle <strong>Expected Volume</strong> on, then click the <strong>Settings</strong> tab.
    </p>
    <img src="/public/images/sla_empty.png" alt="SLA Empty" />
    <ul>
      <li>If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.</li>
      <li>In the <strong>Expect at least</strong> portion of the window:
        <ul>
          <li>Set the minimum number of transactions you expect to be processed (the volume)</li>
          <li>Use the <strong>Every</strong> fields to specify the time frame</li>
          <li>Indicate when the SLA should go into effect. If you choose <strong>Starting on</strong>, complete the date and time fields.</li>
          <li>Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose <strong>Everyday</strong> if necessary.</li>
        </ul>
      </li>
      <li>In the <strong>Set status to 'At Risk'</strong> portion of the window, specify when the SLA should be marked as at risk.
        <ul>
          <li>By default, notifications are not sent until an SLA is in violation. To change that, check <strong>Send an 'At Risk' notification</strong>.</li>
        </ul>
      </li>
    </ul>
    <p>
      The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received.
    </p>
    <img src="/public/images/sla_defined.png" alt="SLA Configuration Example" />
    <Note>
      You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click <strong>Settings</strong> on the navbar, then navigate to <strong>Alerts &gt; General Alerts</strong>. Click the tablet and pencil icon to edit, and uncheck the <strong>SLA Alerts</strong> setting.
    </Note>
  </>;

export const AlertsTab = ({siteNameShort = "Arc"}) => <>
    <p><em>Settings related to configuring alerts.</em></p>
    <p>
      Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, {siteNameShort} uses the global settings on the <a href="/26.3/cloud/en/getting-started/administration/settings/alerts">Alerts</a> tab. To use other settings for this connector, toggle <strong>Override global setting</strong> on.
    </p>
    <p>
      By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the <strong>Enable</strong> checkbox.
    </p>
    <p>
      Enter a <strong>Subject</strong> (mandatory). Check <strong>Allow {siteNameShort}Script in Subject</strong> to use {siteNameShort}Script in the <strong>Subject</strong> field. When you select this, the <strong>{siteNameShort}Script Editor</strong> button appears (<img src="/public/images/rest_arcscript_editor.png" alt="arcscript editor button" style={{
  display: 'inline',
  verticalAlign: 'middle',
  margin: 0
}} />).
    </p>
    <p>
      Optionally, enter a comma-separated list of <strong>Recipient</strong> emails.
    </p>
  </>;

export const MiscConnector = () => <>
    <p><em>Miscellaneous settings are for specific use cases.</em></p>
    <table>
      <thead>
        <tr>
          <th>Setting</th>
          <th>Description</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><strong>Other Settings</strong></td>
          <td>Enables you to configure hidden connector settings in a semicolon-separated list (for example, <code>setting1=value1;setting2=value2</code>). Normal connector use cases and functionality should not require the use of these settings.</td>
        </tr>
      </tbody>
    </table>
  </>;

export const Logging = () => <>
    <p><em>Settings that govern the creation and storage of logs.</em></p>
    <table>
      <thead>
        <tr>
          <th>Setting</th>
          <th>Description</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><strong>Log Level</strong></td>
          <td>The verbosity of logs generated by the connector. When you request support, set this to <strong>Debug</strong>.</td>
        </tr>
        <tr>
          <td><strong>Log Subfolder Scheme</strong></td>
          <td>Instructs the connector to group files in the Logs folder according to the selected interval. The <strong>Weekly</strong> option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.</td>
        </tr>
        <tr>
          <td><strong>Log Messages</strong></td>
          <td>Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the <strong>Transactions</strong> tab.</td>
        </tr>
      </tbody>
    </table>
  </>;

export const MacrosExamples = ({extraMacros = []}) => <>
    <p>
      Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All
      macros that take an argument use the following syntax: <code>%Macro:argument%</code>
    </p>

    <p>Here are some examples of the macros that take an argument:</p>

    <ul>
      <li>%Header:headername%: Where <code>headername</code> is the name of a header on a message.</li>
      <li>%Header:mycustomheader% resolves to the value of the <code>mycustomheader</code> header set on the input message.</li>
      <li>%Header:ponum% resolves to the value of the <code>ponum</code> header set on the input message.</li>
      <li>%RegexFilename:pattern%: Where <code>pattern</code> is a regex pattern. For example, <code>%RegexFilename:^([\w][A-Za-z]+)%</code> matches and resolves to the first word in the filename and is case insensitive (<code>test_file.xml</code> resolves to <code>test</code>).</li>
      <li>%Vault:vaultitem%: Where <code>vaultitem</code> is the name of an item in the <a href="/26.3/cloud/en/getting-started/administration/settings/global-settings-vault">vault</a>. For example, <code>%Vault:companyname%</code> resolves to the value of the <code>companyname</code> item stored in the vault.</li>
      <li>%DateFormat:format%: Where <code>format</code> is an accepted date format (see <a href="/26.3/cloud/en/scripting/value-formatters/date-formatters#sample-date-formats">Sample Date Formats</a> for details). For example, <code>%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%</code> resolves to the date and timestamp on the file.</li>
      {extraMacros.filter(item => item.example).map(item => <li key={`ex-${item.name}`}>{item.example}</li>)}
    </ul>

    <p>You can also create more sophisticated macros, as shown in the following examples:</p>

    <ul>
      <li>Combining multiple macros in one filename: <code>%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%</code></li>
      <li>Including text outside of the macro: <code>MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%</code></li>
      <li>Including text within the macro: <code>%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%</code></li>
    </ul>
  </>;

export const MacrosTable = ({siteName = "CData Arc", extraMacros = []}) => <>
    <p>
      Using macros in file naming strategies can enhance organizational efficiency and contextual
      understanding of data. By incorporating macros into filenames, you can dynamically include
      relevant information such as identifiers, timestamps, and header information, providing
      valuable context to each file.
    </p>

    <p>{siteName} supports these macros, which all use the following syntax: <code>%Macro%</code>.</p>

    <table>
      <thead>
        <tr><th>Macro</th><th>Description</th></tr>
      </thead>
      <tbody>
        <tr><td>ConnectorID</td><td>Evaluates to the ConnectorID of the connector.</td></tr>
        <tr><td>ConnectorName</td><td>Evaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.</td></tr>
        <tr><td>Ext</td><td>Evaluates to the file extension of the file currently being processed by the connector.</td></tr>
        <tr><td>Filename</td><td>Evaluates to the filename (extension included) of the file currently being processed by the connector.</td></tr>
        <tr><td>FilenameNoExt</td><td>Evaluates to the filename (without the extension) of the file currently being processed by the connector.</td></tr>
        <tr><td>MessageId</td><td>Evaluates to the MessageId of the message being output by the connector.</td></tr>
        <tr><td>RegexFilename:<em>pattern</em></td><td>Applies a RegEx pattern to the filename of the file currently being processed by the connector.</td></tr>
        <tr><td>Header:<em>headername</em></td><td>Evaluates to the value of a targeted header (<code>headername</code>) on the current message being processed by the connector.</td></tr>
        <tr><td>LongDate</td><td>Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).</td></tr>
        <tr><td>ShortDate</td><td>Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).</td></tr>
        <tr><td>DateFormat:<em>format</em></td><td>Evaluates to the current datetime of the system in the specified format (<code>format</code>). See <a href="/26.3/cloud/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters">Sample Date Formats</a> for the available datetime formats.</td></tr>
        <tr><td>Vault:<em>vaultitem</em></td><td>Evaluates to the value of the specified vault item.</td></tr>
        {extraMacros.map(item => <tr key={item.name}>
            <td>{item.name}</td>
            <td>{item.description}</td>
          </tr>)}
      </tbody>
    </table>
  </>;

export const Performance = () => <>
    <p><em>Settings related to the allocation of resources to the connector.</em></p>
    <table>
      <thead>
        <tr>
          <th>Setting</th>
          <th>Description</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><strong>Max Workers</strong></td>
          <td>The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the <a href="/26.3/cloud/en/getting-started/administration/settings/performance-settings">Performance Settings</a> portion of the <a href="/26.3/cloud/en/getting-started/administration/settings/advanced-settings">Advanced Settings</a> page.</td>
        </tr>
        <tr>
          <td><strong>Max Files</strong></td>
          <td>The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the <a href="/26.3/cloud/en/getting-started/administration/settings/performance-settings">Performance Settings</a> portion of the <a href="/26.3/cloud/en/getting-started/administration/settings/advanced-settings">Advanced Settings</a> page.</td>
        </tr>
      </tbody>
    </table>
  </>;

export const siteNameShort = "Arc";

export const siteName = "CData Arc";

The S3 connector integrates with Amazon's S3 (Simple Storage Service) and other S3-like services (such as Google Storage and Wasabi).

## Key Capabilities

* Amazon S3 and S3-compatible service integration (Google Storage, Wasabi) with IAM role and access key authentication
* Bucket-based file organization with bidirectional transfers and prefix-based virtual folders
* Client-side and server-side encryption options with configurable access policies
* Optional caching to ensure only new or updated files are downloaded

## Overview

Each S3 connector can automatically upload to and download from a single S3 bucket.

Before you begin, you need an Amazon account with the appropriate credentials (or account credentials for the S3-like service you are using). Specify the upload and download paths in the bucket. The connector supports download filters by file name.

## Connector Configuration

This section contains all of the configurable connector properties.

### Settings Tab

#### Host Configuration

*Settings related to the remote connection target.*

| Setting                   | Description                                                                                                                             |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Connector Id**          | The static, unique identifier for the connector.                                                                                        |
| **Connector Type**        | Displays the connector name and a description of what it does.                                                                          |
| **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow.                                         |
| **Service**               | Use the dropdown to choose which service to connect to. Select **Other** to specify the base URL to use when connecting to the service. |
| **Bucket Name**           | The S3 bucket to poll or upload to.                                                                                                     |
| **Region**                | The Region where the specified **Bucket Name** is stored.                                                                               |

#### Account Settings

*Settings related to the account with permission to access the configured **Bucket Name**.*

| Setting             | Description                                                                                                                                                                                                     |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IAM Role**        | Whether to use the attached IAM role to access S3. Only use this setting when {siteName} is hosted on an EC2 instance that has an IAM role attached. The IAM credentials replace the two **Key** options below. |
| **Access Key**      | The Access Key account credential acquired from Amazon (or the S3-like service).                                                                                                                                |
| **Secret Key**      | The Secret Key account credential acquired from Amazon (or the S3-like service).                                                                                                                                |
| **Assume Role ARN** | Use the two **Key** options above to call the Amazon STS service to obtain temporary credentials to access S3 with the provided role ARN.                                                                       |

#### TLS Settings

*Settings related to TLS negotiation with the S3 server.*

| Setting                       | Description                                                                                                                                                                                                                                                                                                                                                                                                        |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **TLS**                       | Check this to enable TLS negotiation.                                                                                                                                                                                                                                                                                                                                                                              |
| **Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the server requires a specific certificate for validation. If the server does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server's identity. |

#### Upload

*Settings related to the path in the specified bucket where files are uploaded.*

| Setting              | Description                                             |
| -------------------- | ------------------------------------------------------- |
| **Prefix**           | The remote path on the server where files are uploaded. |
| **Overwrite Action** | Whether to overwrite, skip, or fail existing files.     |

#### Download

*Settings related to the path in the specified bucket where files are downloaded.*

| Setting         | Description                                                                                                                                                                                                                                                                                                                                      |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Prefix**      | The remote path on the server from where files are downloaded.                                                                                                                                                                                                                                                                                   |
| **File Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. |
| **Delete**      | Check this to delete successfully downloaded files from the remote storage.                                                                                                                                                                                                                                                                      |

#### Caching

*Settings related to caching and comparing files between multiple downloads.*

| Setting                  | Description                                                                                                                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download.                    |
| **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. |

<Note><CommonCacheCleanup /> When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`.</Note>

### Advanced Tab

#### Advanced Settings

*Settings not included in the previous categories.*

| Setting                    | Description                                                                                                                                                                                                                                                                                                  |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Access Policy**          | The access policy set on objects after they are uploaded to the S3 server.                                                                                                                                                                                                                                   |
| **Encryption Password**    | If set, object data is encrypted on the client side before upload, and downloaded objects are automatically decrypted.                                                                                                                                                                                       |
| **Recurse**                | Whether to download files in subfolders of the target remote path.                                                                                                                                                                                                                                           |
| **Local File Scheme**      | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros).                                                                      |
| **Server Side Encryption** | Whether to use server-side AES256 encryption.                                                                                                                                                                                                                                                                |
| **TLS Enabled Protocols**  | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2.                                                                                                                          |
| **Virtual Hosting**        | Whether to use hosted-style or path-style requests when referencing the bucket endpoint.                                                                                                                                                                                                                     |
| **Processing Delay**       | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](../flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. |

#### Proxy Settings

<CommonProxySettings />

#### Logging

<Logging />

#### Miscellaneous

<MiscConnector />

### Automation Tab

#### Automation Settings

*Settings related to the automatic processing of files by the connector.*

| Setting                   | Description                                                                                                                                                                                                       |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Send**                  | Whether files arriving at the connector are automatically uploaded.                                                                                                                                               |
| **Retry Interval**        | The amount of time before a failed upload is retried.                                                                                                                                                             |
| **Max Attempts**          | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connector retries the file indefinitely.           |
| **Receive**               | Whether the connector should automatically poll the remote download path for files to download.                                                                                                                   |
| **Interval**              | The interval between automatic download attempts.                                                                                                                                                                 |
| **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. |
| **Time**                  | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*.                                                                         |
| **Day**                   | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*.                                                                                       |
| **Minutes**               | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*.                                                                                 |
| **Cron Expression**       | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*.                                        |

#### Performance

<Performance />

### Alerts Tab

<AlertsTab />

### SLAs Tab

<SlasTab />

## Establishing a Connection

The requirements for establishing an S3 connection are simple:

* Amazon account credentials (or other S3-like account credentials)
  * **Access Key**
  * **Secret Key**
* A bucket that can be accessed by the above account

For Amazon S3, use [this link](https://aws-portal.amazon.com/gp/aws/securityCredentials) to obtain **Access Key** and **Secret Key** information from Amazon.

Optionally, you can secure the connection with S3 servers with TLS by enabling the **Use TLS** option in the [TLS Settings](#tls-settings) section.

## Uploading

### Upload to Remote Folders

The **Prefix** setting in the [Upload](#upload) section of the **Settings** page specifies the bucket path to upload files to. This allows for the logical separation of files into virtual folders in the same bucket.

<Note>S3 servers do not maintain a real folder structure, and {siteNameShort} uses application logic to present a pseudo folder structure. Slashes in the **Prefix** (`/`, `\\`) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from 'subfolders' in the bucket based on the slashes in the path.</Note>

### Upload Automation

The S3 connector supports automatic upload via the [Automation tab](#automation-tab). When **Upload** automation is enabled, files that reach the **Transactions** tab for the connector are automatically uploaded to the specified **Bucket Name** at the specified **Prefix**.

If a file fails to upload, the application attempts to send it again after the **Retry Interval** has elapsed. This process continues until the **Max Attempts** has been reached, after which the connector raises an error.

## Downloading

### Download from Remote Folders

The **Prefix** setting in the [Download](#download) section of the **Settings** page specifies the bucket path to download files from. This allows for the logical separation of files into virtual folders in the same bucket.

The **File Filter** setting provides a way to only download specific filenames in the specified path.

<Note>S3 servers do not maintain a real folder structure, and {siteNameShort} uses application logic to present a pseudo folder structure. Slashes in the **Prefix** (`/`, `\\`) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from 'subfolders' in the bucket based on the slashes in the path.</Note>

### Download Automation

The S3 connector supports automatic download via the [Automation tab](#automation-tab). When **Download** automation is enabled, the connector automatically polls the remote bucket based on the specified **Download Interval**.

## Macros

<MacrosTable />

### Examples

<MacrosExamples />