# AES Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/aes Configuration and usage guide for the CData Arc AES connector, which natively encrypts and decrypts message content using AES. The AES connector provides native AES encryption and decryption of message content directly within CData Arc. ## Key Capabilities * Native AES encryption and decryption of message content, with no external tools or scripts required * Multiple key sources (password-based key derivation, static key, or header-supplied key) and seven cipher modes * Configurable nonce/IV handling, GCM authentication, password salt, and padding, with interoperability for content encrypted by OpenSSL ## Overview The AES connector performs native AES encryption and decryption of message content, replacing prior workarounds that required invoking OpenSSL through ArcScript and managing external processes. The connector offers two operations: **Encrypt** (the default) and **Decrypt**. You configure how keys, cipher modes, and metadata (such as nonces or IVs and authentication tags) are handled. > [!NOTE] > > The connector can decrypt content originally encrypted with OpenSSL (AES-CBC), provided the same key, IV, and padding configuration are used. Passwords, keys, and sensitive header values are masked in all logs, message details, audit records, debug output, errors, and support packages. In addition, authentication failures never return partially decrypted content. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab The available settings depend on the selected **Operation**, **Key** source, and **Cipher Mode**. #### Configuration *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
Use the **Operation** radio buttons to specify whether the connector **encrypts** (the default) or **decrypts** message content. #### Key *Settings related to the AES key. The available fields depend on the selected **Key** source. The following table shows the settings for **Password-Based Key Derivation***. | Setting | Description | | - | - | | **Key** | How the AES key is provided: **Password-Based Key Derivation** (default), **Static** (a fixed key entered in the settings), or a key read from a message **Header**. | | **Password** | The password used to derive the AES key. Interpreted as UTF-8. | | **Key Size (Bits)** | The size of the derived AES key in bits: 128, 192, or 256 (default). | | **Key Derivation Function** | The PBKDF2 pseudorandom function used to derive the AES key: PBKDF2-HMAC-SHA256 or PBKDF2-HMAC-SHA512. | | **Iterations** | The number of PBKDF2 iterations. Higher values increase resistance to brute-force attacks at the cost of performance. The default is `600000`. | | **Password Salt** | How the PBKDF2 salt is provided: **Auto-Generate** (default), a static value, or read from a message header. | | **Generated Salt Output** | (auto-generated salt) Whether the generated salt is **Included** in the payload or written to an output header. | ##### Non-Default Key Options * If you are using a **Static** **Key**, provide the **Key** as hexadecimal (prefixed with `0x`) or Base64. The decoded key must be 16, 24, or 32 bytes (AES-128, AES-192, or AES-256). * If you are using a **Header**-supplied **Key**, binary values *you* enter accept hex (`0x`) or Base64; binary values *the connector generates* and writes to headers are Base64-encoded. The decoded key must be 16, 24, or 32 bytes (AES-128, AES-192, or AES-256). ##### Non-Default Password Salt Options * If you choose **Static** in **Password Salt**, use the **Password Salt** field to provide a value that decodes to exactly 16 bytes, as `0x`-prefixed hexadecimal or Base64. * If you choose **Header** in **Password Salt**, enter the name of the message header used to read or write the password salt in the **Password Salt Header** field. #### Cipher *Settings related to the cipher mode and nonce/IV handling.* | Setting | Description | | - | - | | **Cipher Mode** | The AES cipher mode: **GCM**, **CBC**, **ECB**, **CFB**, **OFB**, **CTS**, or **CTR**. GCM is the default, and is recommended because it provides both confidentiality and authentication. ECB is provided only for legacy compatibility. The other options provide confidentiality, but not authentication. | | **Padding Mode** | (CBC and ECB modes only) The padding scheme: **PKCS7** (default), **Zeros**, **None**, **ANSIX923**, or **ISO10126**. | | **Nonce / IV** | How the nonce or IV is provided: **Auto-Generate** (default; a unique nonce or IV is generated for each encryption), **Static**, or **Header**. GCM uses a 12-byte nonce; the other modes use a 16-byte IV. ECB does not use a nonce or IV. Selecting **Static** displays a warning that reusing a static nonce or IV with the same key is insecure for GCM and weakens the other modes. If you select **Header**, supply the header name. | | **Generated Nonce / IV Output** | (**Auto-Generate** only) Whether the generated nonce/IV is **Included** in the payload or written to an output **Header**. | > [!NOTE] > > In **Decrypt** mode, the **Nonce / IV\*\*\*\*Auto-Generate** field changes to **Included**. #### Authentication *GCM cipher mode only.* | Setting | Description | | - | - | | **Authentication Tag** | Whether the GCM authentication tag is **Included** in the payload or written to an output **Header**. | | **Authentication Tag Length (Bits)** | The authentication tag length in bits: 128 (the default), 120, 112, 104, or 96. 128 is recommended; shorter tags reduce authentication strength and should be used only for interoperability. | | **Additional Authenticated Data** | Optional Additional Authenticated Data (AAD): **None** (default), or read from a message **Header**. | | **Additional Authenticated Data Header** | The name of the message header used to read the AAD. A value beginning with `0x` is decoded as hexadecimal; otherwise it is interpreted as UTF-8. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send Automation** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Included Payload Format When metadata is included in the payload, the connector orders the fields as follows, omitting any field that is not configured: ``` salt || nonce/IV || ciphertext || authentication tag ``` ## Default Header Names When keys or metadata are read from or written to message headers, the connector uses these default header names: | Purpose | Default header | | - | - | | Password Salt | `Crypto-Password-Salt` | | AES Key | `Crypto-Key` | | Nonce / IV | `Crypto-Nonce` | | Auth Tag | `Crypto-Auth-Tag` | | AAD | `Crypto-AAD` | ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: You can also create more sophisticated macros, as shown in the following examples: # AI Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ai Connect to AI models via a REST API to process messages using configurable prompts and generate structured outputs within your data flows. AI connectors are designed to connect to a number of AI models that expose a REST API. The supported list is [below](./ai#ai-config-tab). ## Key Capabilities * Multi-provider AI integration * Flexible prompt engineering * Customizable AI model selection * Structured output control * Intelligent content processing ## Overview The CData Arc AI connector is a flexible integration tool that connects to any AI model via REST APIs, enabling you to process input messages using configurable system and user prompts to generate structured outputs within your data flows. It supports popular AI services like OpenAI, Gemini, Ollama, and other OpenAI-compatible providers, with full control over API endpoints, models, and parameters. With ArcScript-enabled prompts and customizable output formats, the connector enables use cases such as extracting insights from unstructured feedback, translating content across languages, and transforming text data into actionable structured information. ### Actions This connector can perform any of the three connector action types: • *Trigger* executes a request on a schedule and might produce files to be sent down the flow. This action serves as the starting point of a flow. • *Transform* accepts messages from the flow as input for a request and produces output. This action serves as the middle of a flow. • *Terminal* accepts messages from the flow as input for a request and acts as the terminal point of the flow. ## Connector Configuration This section contains all of the configurable connector properties. ### AI Config Tab If you have not already configured the global AI settings at **Settings > Advanced > AI Settings**, the **AI Config** tab displays a message that you must do this before you can use the connector. Click the **Configure AI Settings** button, and follow the instructions in [AI Settings](../getting-started/administration/settings/ai-settings). If the global settings have already been configured but you want to override them in this connector, click **Edit**. To override the global settings, toggle on **Override global setting**. Now you can select which AI provider you want to use, enter the appropriate connection details and then save the configuration. The connector supports the following providers: * Claude * Deepseek * Gemini * Grok * Ollama (cloud and local) * OpenAI * OpenAI Compatible (any AI provider that implements OpenAI's API) Use the **Edit AI Settings** pane to define the **Base URL** (only available when the provider is **Ollama** or **OpenAI Compatible**), add the provider's **API Key**, and select a **Model**. To populate the **User Prompt** field, you can click **Suggested Prompts** to see a set of predefined prompts for a variety of use cases, or you can enter your own prompt directly into the field. This is where you define the task or request that you want the connector to perform when it receives a file. For example, you could enter something like *Summarize the inbound EDI X12 file using natural language* so that employees who are not familiar with the EDI X12 structure can read the summary and understand what information the document contains. This field supports ArcScript. You can also choose to **Attach input messages to requests**. This is especially useful when you are providing binary files (such as a PDF or image) to the AI provider. To include non-binary files in the request, you need to use \_message.body in your prompt, like this: ``` [_message.body] ``` > [!NOTE] > > See the Summarize option in the Suggested Prompts dropdown for a complete example. Toggle **System Prompt** to provide a set of specific instructions, constraints, languages, or roles that the AI provider should use when providing its response. For example: *Do not use any em dashes or emojis in your response.* This field also supports ArcScript. The way in which you test your configuration and prompts varies by the connector's [action type](#actions): * If your connector is a trigger, click the **Test** button. * If your connector is a transform or terminal, on the **Testing** tab, paste content into the **Input Body** or use the dropdown to add **Headers** and values, then click the **Test** button. In both cases, the test results appear in the **Testing** tab. Logs from the test are displayed in the **Log** tab. ### Settings Tab *Settings related to the core configuration of the connector.* #### Configuration | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Action** | The action this connector is performing: trigger, transform, or terminal. | | **Output in JSON** | If your model supports structured outputs, check this to deliver the connector output in JSON format. This is useful when you want to ensure the AI output is structured and machine-readable, so it is easier to parse, extract data, or use in downstream connectors and processes. | #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Timeout** | The AI response timeout, in seconds. By default, this value is set to 60 seconds. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message: No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Options on this tab vary depending on the connector's [action type](#actions): * If your connector is a **Trigger**, use the **Receive Files** button to start the flow. * If your connector is a **Transform** or **Terminal**, use the **Upload Files** button to upload files to the flow. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: You can also create more sophisticated macros, as shown in the following examples: # API Connector Overview Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api Expose database tables, views, and stored procedures as a web API from CData Arc. The API connector exposes tables, views, and stored procedures from a database as a web API. The API Connector reads the available tables, views, and stored procedures from the target database, then any subset of these can be exposed as API resources. ## Key Capabilities * Exposes database tables, views, and stored procedures as RESTful web APIs * Automatic schema generation from database connections * Comprehensive API management with rate limiting and CORS support * Multiple authentication options and user management capabilities * Multiple output formats (JSON, XML, and CSV) The following topics cover all aspects of the API connector: * [Configure a Connector](./api-config) covers all of the configurable connector properties on each tab. * [Establish a Connection](./api-establish-connection) explains how to connect to your data store. * [Build Your First API](./api-build-first-api) explains how to use a wizard to generate schemas for reading and writing to your data store. * [API Resources](./api-resources) describes options related to managing resources. * [API Actions](./api-actions) describes example actions to extend the set of operations on resources and to execute unrelated actions on the server. * [Query Your API](./api-query) covers how to expose your API resources to authorized users and IP addresses. * [API Management](./api-management) covers discovery options, server responses, rate limiting and cross-origin resources sharing (CORS). * [Authentication](./api-authentication) covers user management and authentication options. * [Additional Output Formats](./api-additional-output-formats) describes how to use the other output formats supported by the API connector. > [!NOTE] > > The API connector supports publishing custom APIs at a CData Arc endpoint. If you are looking for an API client to consume APIs, use the [REST Connector](../rest) instead. # API Actions Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-actions Extend API operations with custom actions on the API connector. Use actions to extend the set of operations on resources and to execute unrelated actions on the server. Actions must be executed as an [HTTP POST](./api-resources#post) request. The following is an example action invocation: ``` POST http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/WashCar/ { Wax: "Wax_1" } ``` Here is the corresponding response: ```json theme={null} {"Complete": "Complete_1"} ``` ### Use URL Parameters as Input In cases where the client does not support the HTTP POST method, you can invoke the Action using the URL parameters `@x-http-method`. The inputs for the Action can then be specified as additional URL parameters. For example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/WashCar?@x-http-method=POST&URLparam1=my_url_parameter1&urlparam2=my_url_param2` # Additional Output Formats Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-additional-output-formats XML, JSONP, RSS, HTML, CSV, and TSV output formats for the API connector. In addition to the default JSON format that all resources and actions support, the API also supports the following formats: XML, JSONP, RSS, HTML, CSV, and TSV. ## XML Request the XML (OData Atom) format by adding the `$format=atom` query parameter or by adding the HTTP Accept header with the value `application/xml` to the request. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$format=atom` ### Response ```xml theme={null} Cars http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars/ 2015-04-23T18:36:20-04:00 1000http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars('1000') 1000 Accord ``` ## JSONP Request the JSONP format by adding the `$callback=myCallback` query parameter, where `myCallback` is the name of the function you would like to wrap the JSON result. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$callback=myCallback` ### Response ```json theme={null} myCallback( { "@odata.context": "http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "value": [ { "Id": "1000", "Model": "Accord" }, ... ] } ); ``` ## RSS Request the RSS format by adding the query parameter `@rss` to the request. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?@rss` ### Response ```xml theme={null} Retrieves and updates Cars information Arc - http://arc.cdata.com http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?@html Cars 1000 Accord ``` ## HTML Request the API response to be formatted as a simple, unstyled HTML table by adding the query parameter `@html` to the request. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?@html` ### Response ```html theme={null} ``` ## CSV Request the API response to be formatted as CSV data by adding the query parameter `@csv` to the request. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?@csv` ### Response ```csv theme={null} Id,Model 1000,Accord ``` ## TSV Request the API response to be formatted as TSV data by adding the query parameter `@tsv` to the request. See the example request and response below. ### Request `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?@tsv` ### Response ```tsv theme={null} Id Model 1000 Accord ``` # Authentication Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-authentication User management and authentication options for the API connector. ## Users and Authentication Arc uses OAuth 2.0 and authtoken-based authentication to control per-user access to the API. Authtokens are unique, randomly generated identifiers that users authenticate to an API with. Authtokens also represent the user's current permissions. Authtokens let you connect using any of the major authentication schemes. For example, to use HTTP Basic authentication, set the username in the User header and the user's corresponding authtoken in the Password header. See [Authentication](#authentication) for more information on authenticating to the API. OAuth 2.0 provides a more structured authentication flow for clients that need to manage token lifecycles or integrate with external applications. Arc supports the Client Credentials grant type, which is intended for server-to-server integrations where no user interaction is required. Rather than authenticating directly with a static authtoken, OAuth 2.0 clients obtain a short-lived access token by presenting their client credentials to the Access URL. ### Adding Users To add a user, navigate to the **Users** tab for the connector, and click **Add**. In the **Settings** dialog, enter the following information: | Field | Description | | - | - | | **Name** | Enter a username to be used in HTTP authentication. | | **Active** | Indicate whether the user is active or inactive. | | **Operations** | Select the HTTP methods the user is allowed to access: GET, POST, PUT/PATCH/MERGE, or DELETE. These correspond to SELECT, INSERT, UPDATE, and DELETE statements, respectively. | Click the **Authentication** tab to add details on authtokens and OAuth 2.0. | Field | Description | | - | - | | **Authtoken** | The system-generated authtoken. Make sure to copy it to a safe location before saving the user, because it is not shown again. | | **Maximum Requests per Hour** | Uncheck **Unlimited Requests** to set a limit on the maximum requests per hour for this user. | | **Maximum Concurrent per Hour** | Uncheck **Unlimited Requests** to set a limit on the maximum number of requests that can be sent simultaneously. | | **Grant Type** | The OAuth 2.0 grant type that determines how the client application obtains an access token. **Client Credentials** is currently the only option. | | **Client Id** | The system-generated client Id token. | | **Client Secret** | The system-generated client secret. Make sure to copy it to a safe location before saving the user, because it is not shown again. | | **Access URL** | The endpoint URL used by the client application to request an access token using the configured credentials. | > [!NOTE] > > User-specific settings override server-wide API limit settings. An empty value for one of the user settings uses the server default. ### Configuring the User Database Arc stores user information in an SQLite database by default. You can also save user information to the database of your choice. If you are using the embedded server, specify the connection string to the caching database in the `connectionStrings` element of the Web.config file. Otherwise, refer to the documentation for the server you are hosting the application on. ## Authentication Users access API resources by providing authtokens with requests. You can manage users, authtokens, and OAuth 2.0 credentials on the [Users](./api-config#users-tab) configuration tab. Before users can call the API, you also need to set trusted IP addresses for connections. These settings are available on the [Server](./api-config#users-tab) configuration tab. By default, all IP addresses are restricted. ### Using Authtokens in Basic Authentication Enter the user's authtoken as the password when using Basic Authentication. ### Using Auth Tokens in the HTTP Header Add the HTTP header `x-cdata-authtoken` with the desired authtoken as part of the HTTP request. ### Using Authtokens as Query String Parameters To allow the connector to pass the authtoken in query string parameters, open the **Server** tab of the API connector settings pane and select **Allow authtoken in URL** in the **Advanced Settings** section. After enabling this feature, you can specify the authtoken as the value of the `@authtoken` parameter, which can be supplied as part of the HTTP form-post data or a query parameter. # Build Your First API Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-build-first-api Use the API connector wizard to generate schemas and surface your database resources. After defining a connection to your data source, you can surface the resources of your API. You can use the wizard in the administration console to generate schemas for reading and writing to your data store. ### Surfacing Database Resources To use the wizard to model a database as an API: 1. Click **Resources > Add Resource**. 2. Select a table you want to model as an API. 3. Select the columns you want to surface in the resources. The resource is defined in a schema, which can easily be modified or extended without regenerating from the database. You can modify the schema by clicking the resource or **Edit** in the entry for the resource. The generated schema contains the two essential components of a schema: * An info block containing resource columns definitions * Methods that correspond to HTTP methods and call data processing operations # Configure a Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-config All configurable properties for the API connector in CData Arc. This section contains all of the configurable connector properties. ## Settings Tab ### Connector Details *Settings that determine how the API can be accessed.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **API Endpoint** | The URL to which the custom API is published. To access a specific resource, append a slash and the resource name to this URL value. | | **API Documentation** | The URL to which the documentation for the custom API is published. | ### Connection *Settings related to the database connection. Click **Add Connection** to configure a connection.* | Setting | Description | | - | - | | **Select Connector** | The type of database to connect to. The remaining connection fields vary based on your database choice. The most common fields are described below, but you should review the connector documentation for your database if you need additional information. | | **Server** | The host name or IP address of the server hosting the database. | | **Auth Scheme** | The type of auth scheme to use for authenticating the connection. When you select an auth scheme, additional fields appear. The connector documentation for your database contains details on these fields. Not applicable for DB2. | | **Port** | The port on which to connect to the database host. | | **Database** | The name of the database to connect to. | | **User** | The user credential that has permission to access the database. | | **Password** | The password credential associated with the specified **User**. | ## Tables Tab The **Tables** tab lists all API resources exposed by the connector. Each resource is generated from a table or view present in the connected database. Adding a new table or view exposes new database data to a web API call via the connector. See [API Resources](./api-resources) for details. ## Stored Procedures Tab The **Stored Procedures** tab lists all API actions exposed by the connector. Each action is generated from a stored procedure present in the connected database. Adding a new stored procedure exposes new database data to a web API call via the connector. See [API Actions](./api-actions) for details. ## Users Tab The **Users** tab lists all users authorized to make API calls to the connector. It lets you: * Create users with their associated authtokens and OAuth 2.0 credentials * Define HTTP method privileges (GET, POST, PUT, MERGE, PATCH, and DELETE) * Specify how many requests each user can make per hour * Specify how many concurrent requests are permitted You can add, edit, and delete users on this tab. See [Users and Authentication](./api-authentication#users-and-authentication) for details on each field. > [!NOTE] > > The request settings here override settings in the [Default Rate Limits](#default-rate-limits-per-user) section of the **Server** tab. ## Server Tab ### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
IdModel
1000Accord
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature: > [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. ### Default Rate Limits (Per User) *Settings restricting the number of API requests allowed.* See [Rate Limiting](./api-management#rate-limiting) for more information. | Setting | Description | | - | - | | **Max Requests Per Hour** | The limit to the number of requests a single user can issue in an hour. | | **Max Concurrent Requests** | The limit to the number of concurrent requests a user can issue. | ### Cross-Origin Resource Sharing (CORS) *Settings governing the use of CORS to serve cross-origin resources.*
SettingDescription
**Enable cross-origin resource sharing (CORS)** Whether or not CORS is enabled. The rest of these options are only available when you check this box.
**Allow all domains without '\*'** If enabled, domain origins are not restricted to a specific list.
**Access-Control-Allow-Origin** The comma-delimited list of domain origins to allow. Included as an HTTP response header.
**Access-Control-Allow-Credentials** Whether or not user credentials such as cookies are allowed in cross-origin requests. Included as an HTTP response header.
**Access-Control-Allow-Methods** The comma-delimited list of methods to allow. Included as an HTTP response header.
**Access-Control-Allow-Headers** The comma-delimited list of headers to allow. Included as an HTTP response header.
**Access-Control-Max-Age** The maximum duration (in seconds) that Access-Control response header values can be cached.
### OData *Settings related to OData feature support.* | Setting | Description | | - | - | | **Server-Side Paging Size** | The number of results per-page returned by an OData request. Set this option to 0 to return all results. | | **Default Format** | The OData format to be used when the client does not specify a format. | | **Default Version** | The OData version to be used when the client does not specify a version. | | **Date Time Format** | The format to use when returning date-time information. | | **Base URL** | The fully qualified URL of the OData endpoint. If unspecified, Arc attempts to create the base URL based on the incoming request. | | **Authtoken in URL** | Check this to allow the connector to pass the authtoken in query string parameters for authentication. See [Using Auth Tokens as Query String Parameters](./api-authentication#using-authtokens-as-query-string-parameters) for more information. | ## Advanced Tab ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### Logging | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab. | ### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](../../getting-started/administration/settings/settings-page#alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # Establish a Connection Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-establish-connection Connect CData Arc's API connector to your data source. Before creating your first API, you need to connect to your data source. Arc supports connecting to the following data sources: * SQLite * MySQL * SQL Server * PostgreSQL * Oracle * DB2 * Excel * Azure Data Lake Storage > [!NOTE] > > Data sources referenced in the API connector that require licensing should be licensed on the machine outside of Arc. Once you select the **Data Source Type** on the **Connection** tab, a list of connection properties is generated for the selected data source. Enter your connection credentials and save your changes on this page. Click **Test Connection** to make sure the app is connected to your data source. > [!NOTE] > > The application includes drivers for some common databases, but might not include drivers for all the data sources it supports. For drivers that are not included, make sure to follow the instructions below for your machine type. ## Windows Edition Install the ADO.NET provider on this machine, restart the server, and retry connecting to your data store. ## Java Edition Copy the JDBC driver to the `lib` folder of your Java Web server. # API Management Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-management Manage API discovery, server responses, rate limiting, and CORS for the API connector. You can browse your APIs and manage access to them in the administration console. Click **API** on the navbar to view documentation and examples of accessing the API. Use the settings in the connector to manage resources, users, and server settings for your API. This section contains topics related to managing your API. ## Discovery The application's APIs are based on the OpenAPI specification, and are fully documented and discoverable. The APIs can be accessed from standard JavaScript, applications compliant with OpenAPI, or any application that supports the OData standard. The following sections show how to use these standards to discover the APIs. ### Service Document The service document is a simple listing of all the APIs in JSON format by default. The service document is returned from the service root, where all requests are made, including metadata discovery requests. The following is an example of the service root: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/` To retrieve more detailed information, make a request to the metadata URL. ### Metadata URL The application exposes the capabilities of its APIs to OData consumers through the OData metadata document URL. The metadata document is returned as XML and contains column data types, the key of the resource, and other information. You can access the complete metadata resource by appending `$metadata` to the service root, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata` To access the metadata for a resource, append `$metadata` to the resource URL: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars/$metadata` ### OpenAPI Definition OpenAPI (Swagger) definitions are generated for the data sources you surface through the application. To obtain the Swagger definition, append `$swagger` to the service root, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$swagger` ## Server Responses The following is a list of typical server responses to API resources and descriptions of when to expect such responses. | Response | Description | | - | - | | 200 OK | The request was processed by the server without error. | | 201 Created | The request was successful and the specified resource was created by the server. | | 204 No Content | A request returns this status if the requested resource has a null value or the service applies a return=minimal preference. | | 400 Bad Request | The request was not understood or was missing required parameters. | | 401 Unauthorized | The user was not authenticated or authorized to access this resource. | | 403 Forbidden | Access is denied to this resource. | | 404 Not Found | The resource does not exist. | | 405 Method Not Allowed | The HTTP method specified is not allowed for this resource. | | 429 Too Many Requests | The user has sent too many requests in a given amount of time, or exceeded the maximum amount of concurrent requests. | | 501 Not Implemented | The server does not support the functionality required to fulfill the request. This response is returned when the server does not recognize the request method and is not capable of supporting it for any resource. | ## Rate Limiting You can configure both per-user usage limits and server-wide defaults. See [Default Rate Limits (Per User)](./api-config#default-rate-limits-per-user) for details on how to configure rate limiting. ### User-Specific Limits To configure rate limits for an individual user, navigate to the **User** tab, select a user, and click **Edit**. You can configure the following options: * **Privileges**: Select the HTTP methods the user is allowed to access: GET, POST, PUT/PATCH/MERGE, or DELETE. These correspond to SELECT, INSERT, UPDATE, and DELETE statements, respectively. * **Max. Requests**: Enter the maximum requests per hour for this user. A value of 0 allows the user unlimited access per hour. * **Max. Concurrent Requests**: Enter the maximum number of requests that can be sent simultaneously. A value of 0 allows the user unlimited simultaneous requests. > [!NOTE] > > User-specific rate limits override the server defaults. An empty value for one of the user settings uses the server default. ## CORS You can configure cross-origin resource sharing (CORS) on the **Server** tab. CORS allows browser-based clients to connect to Arc. Without CORS, browser-based scripts cannot connect to Arc because of the same-origin policy enforced by the browser. This policy restricts client-side scripts and documents from loading resources outside their origin. The origin of a script consists of the protocol, host, and port. See [CORS](./api-config#cross-origin-resource-sharing-cors) for details on how to configure CORS. # Query Your API Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-query Expose API resources to authorized users and IP addresses, and query the API connector. After creating resources for your API, you can expose them to authorized users and IP addresses. Arc uses auth token-based authentication and supports the major authentication schemes. Click the **Users** tab to manage auth tokens and permissions for users. To define the IP addresses allowed to access the application, click the **Server** tab. You can now query your API in the browser, as shown in the example query and response below. **Request** `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars` **Response** ```json theme={null} { "@odata.context":"http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "value":[ { "Id": "10001", "Model": "MyModel1", "Color": "Red" }, { "Id": "10002", "Model": "MyModel2", "Color2": "Blue" } ] } ``` ## Overriding HTTP Methods Some clients might not be capable of issuing the correct HTTP method for a specific operation. You can use the `@x-http-method` query string input parameter or the `X-HTTP-Method` HTTP header to override the HTTP method for a request. For example, a client that does not support the HTTP PUT method can include the *X-HTTP-Method: PUT* header with a GET request to issue an update request for a resource. # API Resources Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/api/api-resources HTTP methods, filtering, sorting, and pagination for the API connector. The resources in the Arc API use JSON-formatted OData as the default REST protocol for accessing data. Other Web service formats are supported, including OData (Atom), JSONP, HTML, and CSV. ## HTTP Methods Resources are objects exposed in your API that can be queried, created, updated, and deleted. Resources can support the full range of create, read, update, and delete (CRUD) operations or be restricted to only a few. This section describes the HTTP methods used to perform CRUD operations on resources exposed by the application. ### GET You can use an HTTP GET request to retrieve a resource or a set of resources from the server. The following is an example request for an entire collection: `GET http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars` Here is the corresponding response: ```json theme={null} { "@odata.context": "http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "value": [ { "Id": "Id_1", "Color": "Color_1", "Model": "Model_1"}, { "Id": "Id_2", "Color": "Color_2", "Model": "Model_2"}, { "Id": "Id_3", "Color": "Color_3", "Model": "Model_3"} ] } ``` ### POST You can use an HTTP POST request to create a new resource. The request must contain the inputs required to create the resource. The following is an example request: ``` POST http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars { "Color": "Color_1", "Model": "Model_1" } ``` Here is the corresponding response: ```json theme={null} { "@odata.context":"http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "value": [ { "Color": "Color_1", "Model": "Model_2" } ] } ``` ### PUT You can use an HTTP PUT request to update a resource. The primary key is required. The following is an example request: ``` PUT http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars('Id_1') { "Color": "Color_1", "Model": "Model_1" } ``` Here is the corresponding response: ```json theme={null} { "@odata.context":"http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars/$entity", "Id": "Id_1", "Color": "Color_1", "Model": "Model_1" } ``` ### DELETE You can use an HTTP DELETE request to delete a resource. The primary key is required. The following is an example request: `DELETE http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars/('Id_1')` The response is empty with a `204 No Content` HTTP status line. ## Filtering Resources You can use an HTTP GET request to retrieve all resources, filter resources, sort resources, and restrict the data returned from each resource. The path of the URL specifies the set of resources to retrieve. For example, to retrieve all Cars resources, use the following URL: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars` ### Single Resource To retrieve a single resource, make a request to the URL for that resource. To construct the URL, use the desired resource's primary key. For example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars('1000')` Some resources might have multiple primary keys, which are indexed as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars(Id='1000', Date='2016-07-01')` ### Filtering Client applications can retrieve multiple resources based on filters provided in the request. For example, a filter to retrieve all resources where Make matches 'Honda' would look like this: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$filter=Make eq 'Honda'` The application supports the following logic operators for comparison: | Operator | Description | | - | - | | Eq | Equal | | Ne | Not Equal | | Gt | Greater Than | | Ge | Greater Than or Equal | | Lt | Less Than | | Le | Less Than or Equal | | Not | Negation | You can also use `and` and `or` to combine multiple filters. For example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$filter=Make eq 'Honda' and Date lt '2016-07-01'` You can use the `startswith`, `endswith`, `toupper`, `tolower`, and `contains` functions with the `$filter` query option. For example, the following request returns resources with properties that contain the specified substring: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$filter=contains(Make,'Honda')` ## Selecting Properties To retrieve a subset of properties, use `$select`, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$select=Id,Model` This returns the Id and Model properties for all resources that match the filter in the request. You can also retrieve an individual property value for a single resource, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars('1000')/Model/$value` ## Sorting You can use `$orderby` to sort resources, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$orderby=Model asc, Color desc` This returns the resources sorted by Model (ascending) and then by Color (descending). ## Pagination ### Server-Side The application supports server-side paging. Enable this option in **Settings > Server**. When the page size is greater than 0 and a request returns results larger than the page size, the URL for the next page of results is included in the `@odata.nextlink` attribute of the response. The last page of results does not include this attribute. This URL includes a paging token which remains valid for the next two minutes. For example, the following response has three resources and an @odata.nextLink attribute containing the URL for the next page of records: ```json theme={null} { "@odata.context": "http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "value": [ { "Id": "Id_1", "Color": "Color_1", "Model": "Model_1"}, { "Id": "Id_2", "Color": "Color_2", "Model": "Model_2"}, { "Id": "Id_3", "Color": "Color_3", "Model": "Model_3"} ], "@odata.nextLink":"http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$skiptoken=0f87696b-aa28-4a70-b13d-c86af8338c80" } ``` ### Client-Side Arc also supports client-side paging using `$top`, `$skip`, and `$count`. You can use `$top=n` to include only the first n resources in the result. For example, use the following request to show the top ten Cars resources: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$top=10` You can use `$skip=n` to exclude the first n resources from the result. You can use `$top` with `$skip` to implement client-side paging. `$skip` is always applied before `$top`, regardless of their order in the query. For example, the following two queries retrieve the first 20 resources in two pages: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$top=10` `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$top=10&$skip=10` You can set `$count` to true to return the total number of records in the results. If you are using OData version 2.0 or 3.0, you can set `$inlinecount` to allpages instead. For example, consider the following query: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$top=3&$skip=4&$count=true` This query might return a response like the following: ```json theme={null} { "@odata.context": "http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/$metadata#Cars", "@odata.count": 402, "value": [ { "Id": "Id_1", "Color": "Color_1", "Model": "Model_1"}, { "Id": "Id_2", "Color": "Color_2", "Model": "Model_2"}, { "Id": "Id_3", "Color": "Color_3", "Model": "Model_3"} ], "@odata.nextLink":"http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Account?$skip=7" } ``` The total count that matches the filter is returned in the response along with the single page of results. ### Count-Only You can retrieve just the count for resources matching a particular filter with the query, as shown in the following example: `http://MyServer:MyPort/connector/MyAPIPortName/api.rsc/Cars?$count=true&$filter=Make eq 'Honda'` The response is a raw count of resources matching the filter in the request. # Batch Create Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/batch-create Group individual messages into Batch Groups for collective processing using the CData Arc Batch Create connector. In CData Arc, data travels through flows as Messages that consist of a body and metadata headers. For example, if one connector passes a file to another, the file is transmitted as a Message that contains the file in the body and metadata about the file and associated connectors as headers. Messages can be grouped together in Batch Groups, which allow multiple messages to travel together as a single unit. Some uses for Batch Groups include moving data between systems and processing multiple purchase orders at once. When a Message is part of a Batch Group, it is referred to as a Batch Message. ## Key Capabilities * Intelligent message grouping based on correlation IDs and batch size limits * Configurable minimum and maximum batch sizes for optimal processing * Scheduled and immediate batch creation modes * Support for custom correlation headers for advanced grouping logic ## Overview The Batch Create connector produces Batch Groups based on Messages that arrive in the Send folder (the **Transactions** tab of the connector settings pane). When a Batch Group is created, it is passed down the flow through the **Transactions** tab, where it can be processed by other connectors as needed. > [!NOTE] > > The Batch Create connector can only accept individual messages as inputs. If the connector receives an existing Batch Group as an input, it throws an error message. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings that determine how the connector can be accessed.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings *Settings that specify Batch Grouping options.* | Setting | Description | | - | - | | **Correlation Id Header** | The connector looks for headers that match the header name you enter, then groups messages according to the values they have assigned to that header. See [Correlation Id Header Example](#correlation-id-header-example) for details. | | **Max Batch Size** | The maximum number of messages to include in a Batch Group. Any messages in excess of this number wait until the next sending interval. For example, if this is set to 5, and seven messages arrive in the send folder, the connector creates one Batch Group of five messages and adds the final two to the next scheduled Batch Group. | | **Min Batch Size** | The minimum number of messages that must be present to create a Batch Group. For example, if this is set to five, and three messages arrive in the Send folder, the connector does not create a Batch Group from those messages (even if the specified sending interval passes) until at least two more messages arrive. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the creation and sending of Batch Groups.* | Setting | Description | | - | - | | **Enable automation** | If selected, the connector automatically creates Batch Groups following the settings below. If cleared, the connector does not create Batch Groups until you manually trigger it. | | **Send** | If selected, the connector automatically creates Batch Groups from messages as soon as they arrive in the Send folder (**Transactions** tab) as long as other set criteria are met. | | **Scheduled Send** | If selected, the connector uses a scheduled interval to create Batch Groups from messages that arrive in the Send folder. | | **Send Interval** | The interval at which to create Batch Groups. Only available if **Scheduled Send** is enabled. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message: No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Correlation Id Header Example When you enter a header name in the **Correlation Id Header** field, the connector looks for headers that match the name you enter, then groups messages according to the values they have assigned to that header. For example, the following table displays six incoming messages: | MessageId | Filename | Header | Header Value | | - | - | - | - | | 1 | test1.txt | Batch-Correlation-Id | 001 | | 2 | test2.txt | Batch-Correlation-Id | 001 | | 3 | test3.txt | Batch-Correlation-Id | 002 | | 4 | test4.txt | Batch-Correlation-Id | 002 | | 5 | test5.txt | | | | 6 | test6.txt | | | If you enter **Batch-Correlation-Id** in the **Correlation Id Header** field, the connector creates three Batch Groups of these messages: * The first batch group contains the messages with **MessageId** 1 and 2, since they both have the value `001` for the Batch-Correlation-Id header. * The second batch group contains messages with **MessageId** 3 and 4, since they both have the value `002` for the header. * The third batch group contains messages 5 and 6, since neither of them has the Batch-Correlation-Id header. # Batch Merge Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/batch-merge Merges Batch Groups into a single unified message, with XPath-based merging for XML content and flexible header handling options. In CData Arc, data travels through flows as Messages that consist of a body and metadata headers. For example, if one connector passes a file to another, the file is transmitted as a Message that contains the file in the body and metadata about the file and associated connectors as headers. Messages can be grouped together in Batch Groups, which allow multiple messages to travel together as a single unit. Some uses for Batch Groups include moving data between systems and processing multiple purchase orders at once. When a Message is part of a Batch Group, it is referred to as a Batch Message. ## Key Capabilities * Consolidates batch groups into a single unified message * XPath-based merging for XML content with configurable output headers * Flexible header handling (accumulate, keep same, or none) ## Overview The Batch Merge connector accepts a Batch Group that arrives in the Send folder (the **Transactions** tab of the connector settings pane) and merges the Batch Messages in the Batch Group into a single Message. The output Message passes down the flow through the **Transactions** tab, and it can be processed by other connectors as needed. > [!NOTE] > > The Batch Merge connector can only accept Batch Groups as inputs. If the connector receives an individual message as an input, it throws an error message. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings that determine how the connector can be accessed.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **XPath** | Which repeated XML element to use when merging Batch Messages. See [XPath Usage Example](#xpath-usage-example) for more details. | | **Output Headers** | Defines how headers should be transferred from input Batch Messages into the output Message. The options are: **None**: Headers are not kept or merged **Accumulate**: Headers from all batch messages are accumulated, which allows for duplicates or variations **KeepSame**: Only headers that are the same across all batch messages are retained | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings | Setting | Description | | - | - | | **Send** | When checked, the connector automatically merges Batch Groups as soon as they arrive in the Send folder (the **Transactions** tab). If you uncheck this, you must manually instruct the connector to merge Batch Groups. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## XPath Usage Example The example below shows a simple XML tree of a Batch Message: ```xml theme={null} Paper 100 Yes Eraser Bulk 20 No ``` The example below shows a similar, separate Batch Message in the same Batch Group: ```xml theme={null} Pencil, Box 100 10 Yes ``` If you set the value for **XPath** to */Orders/Shipping*, the output message would merge like this: ```xml theme={null} Paper 100 Yes Eraser Bulk 20 No Pencil, Box 100 10 Yes ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: You can also create more sophisticated macros, as shown in the following examples: # Batch Split Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/batch-split Splits Batch Groups into individual messages for distributed processing, with automatic correlation ID assignment for tracking related messages. In CData Arc, data travels through flows as Messages that consist of a body and metadata headers. For example, if one connector passes a file to another, the file is transmitted as a Message that contains the file in the body and metadata about the file and associated connectors as headers. Messages can be grouped together in Batch Groups, which allow multiple messages to travel together as a single unit. Some uses for Batch Groups include moving data between systems and processing multiple purchase orders at once. When a Message is part of a Batch Group, it is referred to as a Batch Message. ## Key Capabilities * Splits batch groups into individual messages for distributed processing * Automatic correlation ID assignment for tracking related messages * Preserves message relationships after split ## Overview The Batch Split connector splits Batch Groups that arrive in the Send folder (the **Transactions** tab of the connector settings pane) into individual messages. When a Batch Group is split, the individual messages are passed down the flow through the **Transactions** tab, where they can be processed by other connectors as needed. > [!NOTE] > > The Batch Split connector can only accept Batch Groups as inputs. If the connector receives an individual message as an input, it throws an error message. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings that determine how the connector can be accessed.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings *Settings that specify Batch Grouping options.* | Setting | Description | | - | - | | **Correlation Id Header** | When the connector splits a Batch Group, it generates a value for the header based on the Batch Group MessageId, then assigns the same header and value to each individual message. If a message already has a Correlation Id Header and value, it does not receive the header and value that is assigned to the rest of the messages. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings | Setting | Description | | - | - | | **Send** | When checked, the connector automatically splits Batch Groups as soon as they arrive in the Send folder (**Transactions** tab). If you uncheck this, you must manually instruct the connector to split Batch Groups. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # Branch Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/branch Routes messages along different flow paths based on configurable conditions, supporting filename, header, XPath, and EDI metadata comparisons. A Branch connector functions as a fork in the flow. Files are sent along different paths in the flow depending on whether a condition specified in the connector is true. ## Key Capabilities * Intelligent flow routing based on configurable conditions (filename, headers, XPath, and EDI metadata) * Support for multiple data types (such as string, number, DateTime) with flexible comparison operators * Advanced XPath evaluation for XML document routing * EDI-specific metadata branching for document type routing ## Overview Branch conditions can check for these different types of values: * the filename * a header on the message * all matching XPaths * the first matching XPath * one or more matching XPaths * [EDI](./edi-connectors) metadata **Notes**: * To use the XPath options, the file being evaluated must be an XML file. * EDI metadata branching can only be used on EDI documents. The Branch connector configuration determines which value to look for, and the *target* values to compare against. If an incoming value matches a target value, the message is passed along a dotted blue flow path. You can have multiple target values, each of which sends the message along a different flow path. If the values do not match any of the conditions, the message is passed along the dotted gray path. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_flow.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=a782014463f895d780964ea005ef7126) See [Branching on Message Headers](#branching-on-message-headers) for a detailed explanation of the image above. ## Connector Configuration This section contains all of the configurable connector properties. ### Branch Config Tab *Settings related to configuring the branch conditions.* The branch settings allow you to specify multiple conditions for a single source value to define your branches.
Setting Description
**Sample File** If you upload a sample XML file, the connector shows the document structure when you use an XPath for comparison. An example is shown below.
**If** Use the dropdown to specify what attribute to compare when messages are received. Options are: the message header, the filename, all matching XPaths, the first matching XPath, one or more matching XPaths, or EDI metadata. Once you make your selection, use the fields to the right to further define the condition, as shown in the **Sample File** image above.
**Data Type** Whether the target value is a string, number, or DateTime. If your data type is **String**, use the **Trim string** and **Ignore case** checkboxes to further refine the comparison. If your data type is **DateTime**, use the field to the right to specify how your DateTime values are structured.
**Trim String** Whether the string comparison should ignore whitespace.
**Ignore Case** Whether the string comparison should be case-sensitive. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_sample_file.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=dfcc058793826c0ca032c840c6bd9c2a) The center portion of the branch settings section (shown in the image below) allows you to set as many conditions as you need. Hover above or below the white pane and select **Add Case** to add conditions. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_conditions.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=6bed3bd57e88b22ece96fb037c4a9dcf) Each condition is made up of three components:
  • the operator (for example, *is equal to, contains, starts with,* or *matches regex*)
  • the value to compare with the value found in the file
  • the *Then output to* component
As you create your flow, connect a different connector to each case statement in the Branch connector to establish the *Then output to* connectors. Finally, connect a different connector to the **Else** case to tell Arc what to do if none of the other conditions are satisfied.
**Ignore Missing Values** Whether to treat missing headers or XPaths as an empty string (not applicable when your **If** selection is *Filename*). This avoids errors being thrown when an expected header or XPath is missing, and instead passes the message along the 'does not match' Flow path.
### Settings Tab *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png) The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Examples Using the Branch Connector The following examples describe common Branch connector use cases. ### Branching on Message Headers In this example, your business receives multiple types of EDI documents from a trading partner, and you need the different types to be processed differently. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_flow_input.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=5fdc16a50786d69faadc716a6daaef7a) The image above illustrates a situation where you send a trading partner purchase orders. They then send you two different types of EDI documents: one is an 855 purchase order acknowledgement and one is an 810 invoice. The [X12 connector](./x12) before the Branch connector in the flow appends **X12-TransactionCode** to each message header. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_flow.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=a782014463f895d780964ea005ef7126) The Branch connector uses the transaction code in the message header to tell Arc what to do when it receives an 810 or an 855 EDI document, and what to do if it receives a document that does not have either of those transaction codes. * When an 810 document is received, the next step in the flow sends it to the invoices table of the back-end system (PostgreSQL in this example.) * When an 855 document is received, the next step in the flow sends it to a PO acknowledgement table in the back-end system. * When any other type of document is received, the **Else** condition uses a [Notify connector](./notify) to send an email to an administrator for manual processing. ### Branching on EDI Metadata In this example, your business receives X12 documents from multiple companies, and you need the documents from two companies to be sent to specific [Workspaces](../flows/workspaces) for further processing. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_edimetadata.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=0a90a67a5491dd715c69fef7df03390c) The image above illustrates a situation where two companies send you X12 documents. The Branch connector uses the X12 Metadata **InterchangeSenderId** to specify where the documents should be delivered. * When a document is received from CompanyA, the next step in the flow sends it to the CompanyA workspace. * When a document is received from CompanyB, the next step in the flow sends it to the CompanyB workspace. * When any other type of document is received, the **Else** condition uses a [Notify connector](./notify) to send an email to an administrator alerting them that an inbound X12 document did not match one of the two expected companies. The company-specific workspaces might then contain a [Workspace Receive](./workspace-receive) connector to receive incoming documents (required because the [Workspace Send](./workspace-send) connector was used in the Branch flow), an [X12 connector](./x12) with identifiers that enable translation of the raw X12 data into XML, an [XML Map](./xml-map/xml-map) connector to handle mapping the XML to the destination, and finally a destination connector ([PostgreSQL](./postgresql/postgresql) in the following image). ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/branch_edimetadata_output.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=d551b980514ffb2ea81482740c0ac7a5) # Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/connectors Connectors are the modular building blocks used to create automated data processing flows in Arc. Connectors are the building blocks for creating a process flow of data through Arc. Each connector performs one of the following tasks: * Transfers files to and from external destinations using a standard network protocol (AS2, AS4, FTP, SFTP, OFTP, etc) * Integrates with a back-end system like a database or cloud application * Consumes or exposes a RESTful web API * Transforms data from one format to another * Compresses/decompresses or encrypts/decrypts data * Modifies the content or headers of a file * Routes files using conditional logic to create a sophisticated data flow Connectors are connected together to create an automatic data-processing flow. After each connector performs its specific task, any files received or processed are passed along to the next connector in the flow. This modular approach allows Arc to handle business tasks with a wide range of complexity. The following image displays how a set of Arc connectors can generate outbound EDI documents using data retrieved as XML from a back-end data source: ![Connectors](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/connectors.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=b8110afc16ec335bcd750604575bfbe4) ### Configuration Each connector has a series of tabs that you use to configure settings for the tasks the connector performs. The tabs include: * **Settings** to manage connection and authentication parameters * **Advanced** for additional connector-specific settings and message and logging information * **Automation** to specify how the connector should automatically process files, poll for files to download, retry failed messages, and so on * **Alerts** if you need to override the global [alert settings](../getting-started/administration/settings/alerts) * **SLAs** to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met * **Transactions** that govern where the connectors interface with the local file system * **Events** scripts for executing custom logic at certain points during connector processing Some connectors, like AS2 and AS4, also require you to configure a local Profile to receive inbound connections. ### Logging When a connector sends, receives, or processes a file, it logs this operation as a transaction. You can view transactions for a specific connector in the connector's **Transactions** tab, and you can view transactions for *all* connectors on the **Transactions** tab of the Activity page. See the Activity documentation for details on what information is captured for each transaction entry. > [!NOTE] > > Date/Time entries are always localized to the local time zone and date format settings of your browser, so even when you browse to a server hosted in a different time zone, the displayed timestamps reflect the local time. ### Messages When connectors process files, they can optionally add metadata to the file. This information is captured as Messages, and it is stored with a special .eml file extension in the application database. See the [Messages Tab](../getting-started/administration/activity#messages-tab) documentation for details on what information is captured on each Message. ### Categories Connectors are grouped into categories: Core, MFT, EDI, Database, and Other. Every connector belongs to one of these categories. * [Core connectors](./core-connectors) provide the foundational functionality for building data workflows. * [MFT connectors](./mft-connectors) extend Arc's ability to send and receive files using secure file transfer protocols. * [EDI connectors](./edi-connectors) support transforming and manipulating EDI documents to integrate these documents with back-end systems or other aspects of a Arc flow. * [Database connectors](./database-connectors) provide the ability to connect to a variety of databases. * [Other connectors](./other-connectors) include various connectors that don't fit into the other categories. They are licensed separately from the rest of the application. In addition to the categories above, each connector in a flow is also assigned one of three behavior categories: Trigger, Transform and Terminal. The configuration of the connector's actions and its location in the flow determine its category. * Trigger connectors start a flow by **Receive Automation** pulling or creating a file or, for connectors that support passive receive, when the connector receives a file from an external client. Examples include [AS2](./as2), [Email Receive](./email-receive), or [SFTP](./sftp). * Transform connectors are connectors that reside in the middle of a flow and are responsible for performing work on the message (modifying it in some way). Examples include [X12](./x12), [XML Map](./xml-map/xml-map), or [CSV](./csv). * Terminal connectors are the end of a flow, where Arc hands the message off to the underlying disk or where the message is consumed by the connector during its send action and no output is generated. Examples include [File](./file), [MySQL](./mysql/mysql), or any database connector configured for Upsert. > [!NOTE] > > * Many connectors, such as [File](./file), [FTP](./ftp), and [SOAP](./soap) can serve as either trigger or terminal connectors, depending on the configured [action](../flows/designing-a-flow#specifying-actions) and where in the flow they appear. > * The [Script](./script), [REST](./rest), and any of the [Database](./database-connectors) connectors can serve as any of the three, again depending on the configured action and where in the flow they appear. ### Bulk Editing Connectors The **Connector View** is accessed by clicking Flows on the navbar, then selecting **Connector View** on the top right. It displays a list of all connectors in your application, enabling users to efficiently locate and manage connectors at scale. If you have connector management permissions, you can use this view to perform bulk edits on connectors. See [Connector View and Bulk Editing Connectors](./bulk-edit-connectors) for details. ### Connector Backoff When a connector repeatedly fails to send files, Arc automatically enters a backoff state, temporarily pausing automation to prevent resource exhaustion and excessive logging. The connector displays a status indicator when it is in backoff, showing the reason and when automation is scheduled to resume. See [Connector Backoff](./connector-backoff) for details. # Core Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/core-connectors Core connectors provide foundational data integration capabilities including file transfer, format conversion, scheduling, and scripting. CData Arc core connectors encompass a broad range of functionality: * Connect to remote servers using common network protocols like FTP, SFTP, and SMTP * Convert flat file formats like CSV (Comma-Delimited Values), TSV (Tab-Delimited Values), and PSV (Pipe-Delimited Values) into XML * Transform XML structures with the XML Map connector's visual designer * Schedule automatic file processing * Expose database resources through a web API * Execute arbitrary business logic through custom scripting These connectors serve as the foundation for data integration workflows. These features are augmented by the [MFT connectors](./mft-connectors) for extensive secure file transfer and the [EDI connectors](./edi-connectors) for automatic EDI document processing. # Common Flows Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/common-flows Examples and guidance for building simple A2A and B2B EDI message flows in CData Arc. When designing your flow to encompass business processes with multiple steps, it is helpful to design and create separate paths in your flow to accommodate different business processes and connect them to the larger flow as you go. This section discusses some use cases for common processing paths that you can incorporate into your flow. ### Video Resources Watch this video to see an example of EDI mapping and integration in Arc. This is part three of a three-part video series that covers each step of an end-to-end B2B integration: managed file transfer (using AS2 as an example), back-end integration, and EDI translation and mapping. Links to the other parts of the series are below. [https://www.youtube.com/embed/wk92g8\_Jmn0](https://www.youtube.com/embed/wk92g8_Jmn0) * Part One: [AS2 Setup in Minutes with CData Arc](https://youtu.be/YjMpP70fczo) * Part Two: [Back-End Integration in CData Arc—Applications and Databases](https://youtu.be/uhWYwsXRc3Q) ## Creating a Simple Flow Application to Application (A2A) flows involve the communication of data between two application data sources for data migration, synchronization, and reporting. ![Simple A2A flow canvas](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/common_flows_1.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=6f4acc9022bbb385725ca5ba24be7f54) A2A flows are relatively simple to configure. To begin, add the trigger connector for the data source to your flow canvas (see [Designing a Flow](./designing-a-flow) for more information). If you do not see the data source you are looking for, check the list of connectors online to see if the data source is available on the [Application and Data Connectors](https://arc.cdata.com/apps/) web page. This connector serves as the source of data in this flow. Next, add the [XML Map Connector](../connectors/xml-map/xml-map) to the canvas. This connector serves as a document map, mapping the data from the table schema in the source connector to the table schema in the destination connector. Then drag from the arrow on the first connector and drop the link on the XML Map connector. Finally, add the terminal connector that corresponds to the integration data source ([Shopify](../connectors/shopify/shopify) in the example above), and link it to the flow. From here, you can configure the connection details of the connectors in the flow by clicking each connector to open its **Settings** tab. CData recommends that you configure the connection details for the source and destination connectors first. When you select the tables that you need, the table schemas are available for selection in the XML Map connector so you can use the connector configuration designer to map the fields from one table to the other table. ## Creating an EDI Message Flow Business to Business (B2B) flows involve the communication of data between two business partners, which is commonly referred to as electronic data interchange (EDI) communications. These flows can be broadly categorized into inbound and outbound message flows. At one endpoint of the flow is a trading partner that typically requires communication over a secure transport protocol, and uses a formal document standard, and on the other endpoint of the flow, integration with the local business logic, either through direct integration with an existing database or ERP system, or as a file in a recognized format for use with an existing back-end solution. ![B2B EDI message flow canvas](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/common_flows_2.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=8e1ac30d06a29ff7f9afa8884c30fd49) B2B flows are more complex to configure than A2A flows, but they are still relatively simple when broken down into their components. While A2A flows usually consist of three connectors (two data sources and a map between them), B2B message flows commonly consist of four connectors (described in the sections below). Depending on your requirements, you might not need all of the connectors to create your B2B flow. To omit any of the connectors shown above, simply proceed to the next connector in your flow and connect it as necessary. If you make a mistake, it is simple to add or remove connectors from the flow. ### Secure Transport In most cases, you need a Secure Transport connector for communication with your trading partner. Common communication protocols include AS2, SFTP, FTP/S OFTP, and AS4, but some industries might adopt other communication protocols. ### Translation/Validation If your trading partner exchanges messages in EDI-X12 or EDIFACT format, you need a corresponding connector of that type to perform both translation and validation for that format. X12 and EDIFACT connectors perform translation to XML for document mapping, and they manage document validation and functional acknowledgment generation and tracking (these are known as 997/999 documents in EDI-X12 and CONTRL documents in EDIFACT). ### Data Mapping If the document standard used by your trading partner is not supported natively in your backend business logic, you need a Data Mapping connector to convert the document from the format that your partner communicates with into one that can be processed in your backend system. The [XML Map Connector](../connectors/xml-map/xml-map) is commonly used for XML mapping when integrating to a database or ERP solution. This restructures the XML to correspond to the document schema of the backend data table. You can also use the [CSV Map Connector](../connectors/map) for structured data formats such as position-based flat files, CSV, PSV, and TSV. Additional Data Mapping connectors include the [PDF Label](../connectors/pdf-label) and [Excel](../connectors/excel) connectors, which are used to render data as a PDF or Excel spreadsheet. These are useful for documents which are exchanged less frequently, and that require manual review in a user-friendly format. ### Integration Finally, to integrate the data directly into a backend database or ERP system, you need an Integration connector to insert the document into the data source. Arc supports a wide variety of database formats, accounting and ERP solutions, and development continues as new systems gain in popularity. # Designing a Flow Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/designing-a-flow How to create, connect, and configure connectors in the CData Arc flow designer. To access the flow designer, click the **Flows** tab, then either create a new [workspace](./workspaces), or click the existing [workspace card](./workspaces#workspace-cards) you want to work in. A flow canvas appears. If you have pre-existing connectors, those connectors are shown on the canvas. From here, you can add and configure new connectors, connect and rearrange existing connectors, and configure existing connectors. * Left-click and hold on empty space in the canvas to drag the designer into position for a better view. * Use the zoom options in the lower-left of the canvas to reposition the connectors to suit your needs. > [!NOTE] > > You can also use a mouse to zoom if the mouse has a scroll wheel. ## Creating Connectors You have four ways to add a new connector to the flow: * Click the **Add** button in the top-right portion of the canvas. * Right-click on the canvas and choose **Add Connector**. * Click the plus sign that appears when you hover over the line connecting two existing connectors. * Hover over the input or output arrow of an existing connector, then click the plus sign that appears. Based on the option you choose, you are presented with an add connector menu that lets you pick the connector to add. If you select one of the first two options above, Arc lacks context about what you're setting up, so you're presented with the dropdown that includes all connectors, as well as the sample flows included in all implementations. If you choose one of the latter two options, Arc presents just the connectors that are applicable for the context you are working in. The following image shows the full window. Each row includes the connector name and icon, and the connector category. Hover over a row to see the number of instances remaining with your current license. You can search by connector name, and filter the list by category. ![Add connector](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_add_modal.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=54dc9be7fff3ab9f560e792464e50471) Once you make your selection, the **Add Connector** pane opens. Here you can name the connector and select which action(s) it should perform. For descriptions of all the information presented in the pane, see [Explaining the Add Connector Pane](#explaining-the-add-connector-pane). Depending on your connector choice, the options visible in the pane vary. For example, some connectors do not have any actions, some have multi-select checkboxes, some have single choice radio buttons, and some have single-select dropdowns. See [Specifying Actions](#specifying-actions) for more information. The pane also displays a preview of the connector as it will appear in the flow, and includes a link to the connector-specific documentation. The following image shows an [AS2](../connectors/as2) connector. ![Add connector pane with AS2 settings](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_add_modal2.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1893611bbf4304ead1a64e4e94dbd8af) > [!TIP] > > When you name the connector (in the **Connector Id** field), be sure to choose a name that clearly describes the operation the connector performs. See [Connector Naming Conventions](#connector-naming-conventions) for some best practice recommendations. Click **Create Connector** to create the connector and open its [configuration](../connectors#configuration) tabs. See [Configuring Connectors](#configuring-connectors) for more information. ### Specifying Actions When you create a connector, the **Add Connector** pane asks you to specify what action(s) you want to use for the connector. Multi-action [MFT connectors](../connectors/mft-connectors) such as [AS2](../connectors/as2) and [AS4](../connectors/as4) support separate send and receive actions within one instance of the connector. Use the checkboxes to select one or both actions, as shown in the following image. ![Connector action checkboxes](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_actions.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=e70e209d566ad7e4a5072c5675318ec5) If you select both the send and receive actions, Arc creates two connector cards, one for each action. The card for the send action has a **Go to Receive Action** link, and the receive card has a **Go to Send Action** link. The following image shows an AS4 connector: ![Multi-action AS4 connector cards](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_multi-action.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=f47b5a312e78f197fdf80ca70833f5b9) > [!NOTE] > > For multi-action connectors, unselected actions do not display in the flow canvas but they are still functional. You can right-click a connector card and choose to show or hide an action, without having to re-create the connector. Single action connectors such as the [Database connectors](../connectors/database-connectors) support one action per instance (for example, Upsert, Select, or Lookup). Use the dropdown to choose your action. Some connectors in the Transform [category](../connectors#categories) do not have selectable actions because they perform one task: translation. The **Actions** portion of the pane does not appear on these connectors. ### Start Triggers and Flow End **Start Triggers** appear before the first connector in a flow. They indicate what actions can initiate the flow. ![Start trigger display](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_start_trigger.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=0c1d2373b96ab1fefaddde7b145d647e) Start triggers can have one or more states depending on which of the following configurations is being used by the connector: * Active receives: Appear when **Receive Automation** is enabled for the connector. The trigger displays *Next scheduled receive*, along with the datetime of the next receive. * Passive receives: Appear when the connector supports passive receiving. This means that external clients can send a file or a request to the connector at anytime. For example, the trigger for an [AS2](../connectors/as2) connector displays *When a file is received* while the trigger for a [Webhook](../connectors/webhook) displays *When a request is received*. * Enable Automation or Receive Manually: Enable Automation works when the user configures **Receive Automation** for a specific time interval. Receive Manually means you need to navigate to the connector Output and click the **Receive** button. This triggers the receive action once. The **Flow End** displays on terminal connectors at the end of a flow, when Arc hands the message off to the underlying disk or where the message is consumed by the connector during its send action and no output is generated. ![Flow end on a terminal connector](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_terminal.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=53c7f22dc9ccd3915901ab0416939003) ## Using Notes If you have permission to modify a flow, you can add notes to your workspaces. Notes can provide clarity regarding the purpose of the flow, and help with organization. Each note includes user names and timestamps detailing when it was created, and when it was last modified. Notes can be placed anywhere on the canvas, and you can drag them to a different position at any time. Because notes are associated with a workspace, they can be [imported and exported](../getting-started/administration/importing-exporting-flows) with their workspace. * To add a note, right-click on a flow canvas and choose **Add Note**. * Give the note a title and provide the body text. * Click **Add Note** to save the note on the canvas. * Use the ellipses on a note to edit or delete it. ![Add note dialog on a flow canvas](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_add_note.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=efae9f012764806b45ef5c683fa56a76) ## Connecting Connectors To rearrange connectors in the designer, drag them to move around the canvas and rearrange them into your desired configuration. To connect connectors, drag from the arrow in the **Output** section of the first connector to the arrow on the left side of the second connector, as shown in the following image: ![Connecting two connectors in the flow designer](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_1.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=cccc1b6e8fc157e91ea0d66a10ea6bce) This creates a logical relationship between the two connectors so that when the first connector generates a message or file, it is passed directly to the second connector for processing. To connect a connector to a different connector, hover over the connection line and click and drag the handlebar at the end of the connection you want to change. Drop the connection onto the new connector. To disconnect a connector, click the handlebars next to the arrows at either end of the connection. ## Configuring Connectors To configure a connector, click the connector on the canvas to open the connector configuration wizard, as shown in the following image: ![Connector configuration wizard](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_2.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=21dd2357c769719893b7231737226071) On the series of tabs that appear, you can configure and test all of the connector settings. When you finish making changes, select the gray x on the left side of the panel to close the connector configuration wizard and return that space to the canvas. ### Additional Configuration Options Right-clicking a connector in a flow brings up the following options: * **Delete Connector** deletes the connector. * **Show Error Path** displays the error path. Use it to specify the connector where files that throw an error should be routed. * **Show Success Path** displays the success path. Use it to specify the connector where files that are successfully sent should be routed. * **Export Connector Settings** exports connector settings from the current server as a zipped file (named `ConnectorId.arcflow`). * **Copy Connector** creates a copy of the configured connector with a new Connector Id. * **Create API Settings** creates a Flow API from the selected connector(s). See [Flow API](./flow-api) for more details. * **Show Receive Action** Only visible on connectors that allow multiple actions but the **Receive** action is currently hidden. Use it to show the receive action for the connector on the canvas. * **Show Send Action** Only visible on connectors that allow multiple actions but the **Send** action is currently hidden. Use it to show the send action for the connector on the canvas. ## Explaining the Add Connector Pane The following image of an AS2 **Add Connector** pane has been annotated to explain what each element in the preview is for. Each number corresponds to a description below. ![Annotated Add Connector pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_add_conn_modal_details.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=cca3198b8773a58c437ef2094c8f867d) 1. Use this arrow to connect the connector with the previous connector in the flow. 2. The connector type. Hover over the icon to see a static description of its intended use. 3. Hover over the information icon to see user-defined information about the connector and its role in the flow. The content displayed comes from the **Connector Description** field on the **Settings** tab. 4. Click the ellipses to open the context menu. It includes actions such as **Show Success Path**, **Show Error Path**, **Copy**, **Delete**, and others. 5. The user-defined name of the connector (from the **Connector Id** field). 6. These five icons show transaction counts. The warning and pending icons are hidden if there are no transactions to display.
IconDescription
![Successful transactions](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_connector_preview_success.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=bccae2d6244c0ba017be2389409ad909) Count of successful transactions.
![Unsent transactions](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_connector_preview_unsent.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=24aefefad4c10e9c495963a185ab4a26) Count of unsent transactions.
![Warning transactions](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_connector_preview_warning.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=e35a0e3cf0fb7eeac0cabb6dddc4eb48) Count of transactions with warnings.
![Error transactions](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_connector_preview_error.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=50be76a48484374338a519ec23c9f6d7) Count of failed transactions.
![Pending transactions](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_connector_preview_pending.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=617b548eaaffcf12cfdda7bc6299bf54) Count of pending transactions.
7. The status bar indicates the status of configured Service Level Agreements (SLAs). SLAs are configured on the connector **Alerts** tab. See [SLAs](../getting-started/administration/activity#slas-tab) for more information. The status bar has four states: * Starting: Displays a date and time in which the SLA will start. * In Progress: Displays when the SLA is in progress and is not at risk. * At Risk: Displays when the SLA is in progress and the configured deadline is approaching, but the configured volume has not been met. * Violated: Displays when the SLA deadline has passed and the expected volume was not met. 8. The gray area in the connector provides key information. For example, an AS2 connector shows the specified action (send and/or receive) and the configured AS2 Id. A database connector shows the specified action (select, upsert, stored procedure, and so on) and the root table. 9. Use the output arrow to connect the connector to the next one in the flow. 10. The path to the connector where files that throw an error should be routed. Only visible if you right-click on the connector and choose **Show Error Path**. 11. The path to the connector where successfully sent files should be routed. Only visible if you right-click on the connector and choose **Show Success Path**. ## Best Practices Most flows have a trigger connector at which data enters the flow and a terminal connector at which data exits the flow. For some cyclical flows, the entry and exit point might be the same connector. For example, data might enter the flow at an AS2 connector when an external trading partner sends a business document over AS2. The data from this business document might need to be imported into a back-end system like a database. In this case, the AS2 connector is the trigger connector and the Database connector is the terminal connector for the flow. When creating a new flow, it is often easiest to start with the trigger and terminal connectors. Then, "work inwards" by adding transform connectors to the flow that fill in the steps between the entry and exit point (for example, you might need to add connectors that transform an inbound AS2 payload into a database insert). ### Interacting with the Local File System CData recommends using the [File connector](../connectors/file) when configuring a flow that interacts with the local file system. This is especially important for folders that are shared with external monitoring processes. Guidelines for [file pickup](#file-pickup) and [file drop-off](#file-drop-off) are outlined below. #### File Pickup To monitor a directory for files to be processed in Arc, add a File connector as the trigger at the beginning of your flow. Uncheck the **Send** action in the **Add Connector** pane. ![File pickup connector configuration](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/file_pickup.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=9962ea0413305b43396968ab4fa39418) Then follow these steps to configure the connector: 1. Open the **Settings** tab. 2. Set the **Path** property to the directory that you want to monitor. You can use macros to dynamically resolve paths based on date and time; hover over the **Path** field for more information. 3. Configure the **File Mask** property with the file pattern that you want to retrieve. 4. To avoid reprocessing files that have already been picked up, configure one of these two options based on what works best for your flow: * In the **Receive** section, check the box for **Delete files (after received)**. This option deletes files after they have been processed. * In the **Caching** section, check one or both caching options to maintain a file size and/or timestamp record of files that have been processed. The application skips cached files but does not delete them. 5. Open the **Automation** tab. 6. **Receive** automation should already be enabled. Set the desired polling interval for the directory. ![Receive automation settings](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/file_pickup_automation.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=dee6c1dec96cf6f51fa3f7cc50424dd8) #### File Drop-Off To drop off processed files to the local file system, add a terminal File connector to the end of your flow. ![File drop-off connector configuration](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/file_dropoff.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=e6849012559be94c1959571da02706ae) Follow these steps to configure the connector: 1. Open the **Settings** tab. 2. Set the **Path** property to the directory where you want to drop off files. You can use macros to dynamically resolve paths based on date and time; hover over the **Path** field for more information. 3. Open the **Advanced** tab. 4. Use the **Overwrite Option** to specify how to handle files that already exist in the destination path. ### Partner-Specific File Transfer Connectors Connectors that send or receive files over the network (for example, AS2, AS4, FTP, SFTP, OFTP, and others) are bi-directional, but are configured for a single trading partner. In other words, a single AS2 connector could send *and* receive AS2 messages with Amazon, but it could not also send and receive files with Walmart. Many MFT (Managed File Transfer) connectors require setting up a profile to exchange files. The [Profiles Page](../getting-started/administration/profiles) contains a tab for configuring application-wide settings for each supported protocol. For example, the information configured in the AS2 profile is used in all AS2 connectors, and partner-specific AS2 connection details are configured in each AS2 connector. ### Connector Naming Conventions Connectors should always be given a descriptive name that communicates the role of the connector in the flow. * The name of an MFT connector (for example, AS2, AS4, FTP, SFTP, OFTP, and others) should indicate the remote party with whom files are exchanged, and the protocol over which the exchange occurs (for example: *Walmart\_Production\_AS2*). * The name of an EDI connector (such as X12 and EDIFACT) should indicate the name of the partner with whom the EDI document is exchanged, the direction of translation, and the format of the document (for example: *Walmart\_X12\_Outbound*). * The name of a Data Mapping connector (such as XML Map) should indicate the format/schemas that the connector is mapping from and mapping to (for example: *Map\_850\_to\_Orders*). * The name of a Back-End Integration connector (for example, Database, MySQL, CData) should indicate the specific back-end system and the table in that back-end system if applicable (for example: *PostgreSQL\_Orders\_table*). If you use the **Connector Description** field on a connector, you can hover your cursor over the information icon to see your description of that connector. If you do not see an information icon, the connector does not have any text in the field. ![Connector description hover tooltip](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/designing_flow_description.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=cc84fcb826ea153028e231c144b25799) ### XML for Data Transformation Arc uses XML as an intermediary format for data manipulation and transformation. Many connectors translate files of various formats into XML and vice versa (for example, the X12 connector, EDIFACT connector, CSV connector, and others). Additionally, Arc uses XML to model back-end system input and output (such as SELECTS and INSERTS into database-like systems). Since most source and destination formats can be represented as XML, data transformation flows often have the following structure: * A connector that pulls or receives data in its original (source) format * A connector that converts this source format into XML (if it is not already XML) * An XML Map connector that transforms this source XML into a new structure representing the target (destination) format * A connector that converts this transformed XML into the destination format (if a non-XML format is required) * A connector that sends the finalized file to its intended destination The [XML Map Connector](../connectors/xml-map/xml-map) is the engine that performs the critical work in these data transformation flows. Becoming comfortable with data transformation is often a matter of familiarizing yourself with the XML Map connector's visual designer. ### Organize Flows into Workspaces You can configure multiple separate flows on the same flow canvas without risk of cross-contamination, but dividing flows into different [Workspaces](./workspaces) can help reduce clutter and maintain organization. Click the **Flows** tab to launch the workspaces grid. From here you can create new workspaces, import or export workspaces, and delete workspaces (except for the Default workspace). Only the connectors from the current workspace are displayed at any given time. Connectors in different workspaces cannot be connected to each other, and the data and configuration files specific to these connectors are held in separate folder structures. # Error Handling Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/error-handling Configure retry, resend, error routing, and global alerts to handle errors in CData Arc flows. Connectors in CData Arc can be configured to flexibly respond to errors that occur during file processing. [Retry](#retry) and [Resend](#resend) settings allow connectors to endure temporary connection issues without interrupting the automated data flow. [Error Routing](#error-routing) in the flow allows for custom processing of files that trigger errors in a connector, and you can configure [Global Alerts](#global-alerts) to notify a system administrator when errors are thrown in the application. ## Video Resources Watch this short video for an introduction to how Arc simplifies error handling. [https://www.youtube.com/embed/qFF7OA88oNM](https://www.youtube.com/embed/qFF7OA88oNM) ## Retry and Resend Many connectors can be configured with retry and resend settings on the **Automation** tab for the connector. These settings govern how the connector responds to a failure to send or upload a file to a remote destination. ### Retry A Retry is triggered when an Arc connector attempts to send or upload a file to a remote server, and the server does not indicate that it successfully received the file. This includes a wide range of potential issues, like basic connectivity problems (for example, a firewall that prevents a connection to the server), permissions issues (for example, the server does not allow access to the destination resource), and application-level processing errors (for example, an AS2 partner's system cannot match the incoming request to a configured AS2 profile). Connectors that support retry logic have **Retry Interval** and **Max Attempts** configuration fields. When a retry is triggered, the connector waits for the **Retry Interval** to elapse, then tries sending or uploading the file again. It continues this process until **Max Attempts** is exhausted. Only once the retries are exhausted does the connector throw an error. The **Max Attempts** value includes the original attempt to send or upload, so setting this value to 1 ensures that a failed send is not retried. Setting this field to a non-positive value instructs the connector to retry indefinitely instead of throwing an error. Utilizing retry logic helps Arc tolerate temporary network conditions that interfere with data transmission. If a partner's server temporarily goes down, or if network conditions are temporarily unstable, the application 'swallows' these errors unless they persist through the entire retry process. This minimizes downtime for non-critical failures and prevents unnecessary error alerts. ### Resend Connectors only support Resend functionality if they implement protocols with built-in acknowledgments (for example AS2 and AS4). A resend is triggered when a connector sends a message that should be acknowledged, but does not receive an acknowledgment in the configured **Resend Interval**. Instead of throwing an error, the connector attempts to send the message again. This process repeats until the connector receives the expected acknowledgement or until the **Max Attempts** has been exhausted. When the resend attempts are exhausted, the connector throws an error. ## Error Routing When a connector encounters a problem sending, uploading, or processing a file, and any available retries and resends have been exhausted, the connector throws an error. Errors specific to a file transaction are logged on the [Activity page](../getting-started/administration/activity), and errors related to more general application resources are logged in the [Application logs](../getting-started/administration/activity#application-logs). The file that caused the error is routed through the flow according to the Error Path. Error paths are represented by a red dotted line that is separate from the blue flow path. Error paths are hidden by default, but you can see them by right-clicking a connector and selecting **Show Error Path**: ![Show error path option in the connector context menu](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/show_error_path.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=adcd44013d80437c79536399b5284eba) Once the error path is visible, drag the red dot to the left side of another connector (just like the blue flow path) to configure the error path. Files that cause an error are routed to the connector that is connected via the error path. Files that do not cause an error are unaffected by the error path. ### Routing to Notify Connectors [Notify connectors](../connectors/notify) are commonly used with error paths. This setup is useful if a system administrator needs to be notified when a connector is unable to send, upload, or process a file. Configure a Notify connector with the target email for the system administrator, then use the error path to connect the connector where the error occurs to the Notify connector. Once configured, emails containing information about the file that raised an error are automatically sent to the specified recipient. > [!NOTE] > > Notify connectors use SMTP settings configured on the [Alerts](../getting-started/administration/settings/settings-page#alerts) tab of the **Settings** page to send emails. ## Global Alerts In addition to routing specific error-causing files with error paths, Arc also supports a global error alert system. The [Alerts](../getting-started/administration/settings/settings-page#alerts) tab of the **Settings** page contains settings for enabling global alerts. Enabling global alerts is functionally equivalent to connecting each connector in the flow to a shared Notify connector via the error path described above. Once enabled, any errors thrown by the connectors in the flow automatically generate an outgoing email (and/or a Windows Event) with information pertaining to the error. # Flow API Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/flow-api Create and expose CData Arc flows as queryable REST endpoints using the Flow API. The Flow API enables you to create modular, single-path Arc flows and expose them as endpoints that you can query. Arc processes data that you pass to Flow API endpoints and returns the results to your querying service. You can use Flow APIs to perform Arc workflows from external applications and services. This flexibility enables you to manage your workflows and automation using your choice of tools. ## Video Resources Watch this short video to see how to setup and use the Flow API. [https://www.youtube.com/embed/Ed25uck0mwo](https://www.youtube.com/embed/Ed25uck0mwo) ## Creating a Flow API Follow these steps to create a Flow API: 1. On a flow canvas, right-click a connector and select **Create API Settings**. ![Create API Settings option in connector context menu](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_create.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=fc5a213a4a3357030ae84b968325ea85) > [!NOTE] > > Flow APIs do not currently support certain connectors. If a connector is not supported, Arc displays an error message when you try to add the connector to a Flow API. See [Unsupported Connectors](#unsupported-connectors) for a list of connectors that cannot be used in Flow APIs. 2. The **Create API Settings** page opens. In the **Method** field, select the HTTP method that you want your Flow API to accept requests for. Options are **GET**, **POST**, **PUT**, **PATCH**, and **DELETE**. You can change this as needed after creating the Flow API. ![Create API Settings page](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_createsettings_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=e010a1839d6786070ccc4687c5ab3a26) 3. In the **Path** field, enter a name for the Flow API. You use this name when issuing queries to the API. > [!NOTE] > > You can only re-use API names if they perform different functions. For example, you can have a Flow API called API\_X12 that performs a POST function and a Flow API called API\_X12 that performs a PATCH function, but you cannot have two Flow APIs called API\_x12 that perform POST functions. 4. Click **Create API**. The **Flow API Settings** opens on the **Settings** tab, and the Flow API appears in the workspace. ![Flow API in the workspace](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_workspace_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=385ec8c893e6af4bd351e06aa01aed88) 5. Drag additional connectors into the Flow API box and connect them together. You can also add notes to the Flow API box. > [!NOTE] > > The first connector in the flow API receives input data from the entity issuing the query, and the final connector sends output data back to the entity issuing the query. When you finish adding and linking your connectors, you can configure and test your Flow API. ### Configuring Settings Click the gear-shaped settings icon in the header of a Flow API to open the **Settings** pane. ![Opening the Flow API settings pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_opensettings.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=1c7f7c173ed50e18f5e0d815179a50d9) In this pane, you can configure the behavior of your Flow API. ![Flow API settings pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_settings_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=100ecbf80f258261ab97b09c8335494c) These settings are outlined in the following sections. #### Request In the **Path** field, select the request type for the Flow API. If you change the request type, be sure that the structure and configuration of your connectors is correct for the new request type. Use the **Description** field to enter an optional description for the Flow API. This is useful when you have multiple Flow APIs and you want to keep track of the purpose of each one. #### Metadata Headers All headers that you include on a request are persisted on the message as it moves through the flow. These headers are prefixed with `API-Header-`, so a header called `MyHeader` would appear as `API-Header-MyHeader`. These headers appear when you expand messages on the [Activity page](../getting-started/administration/activity). You can also see the header information when you expand transactions on the connector **Transactions** tab. #### Query Parameters This section enables you to specify query string parameters for the flow. If you add parameters here, they persist as metadata headers when the message passes down the flow. These headers are prefixed with `API-QueryParameter-`, so a query string of `test` would use `API-QueryParameter-test`. These headers appear in the same places as the [metadata headers](#metadata-headers). This feature is useful if you have information that you want to pass as part of the request, but the information is not part of the body. For example, if you want the Id of the most recent sales order for a specific company in a database, you can add a CompanyName query parameter to your API and reference it within the flow using the `API-QueryParameter-CompanyName` header. #### Body and Response In the **Body** section, choose the **Type** that corresponds to the way you expect the querying client to communicate with the Flow API: * **None**—no body expected with the request * **Raw**—free-form data in any arbitrary format * **Form data**—name-value pairs * **x-www-form-urlencoded** In the text field for **Body** and **Response**, use the drop-down menus to select the expected data formats for the body and response (**XML**, **JSON**, or **Custom**). #### Working with Binary Files The Flow API supports uploading and downloading binary files (PDF, JPEG, or other binary file types) as the request body or the response body. When you open the testing pane (see [Testing Flow APIs](#testing-flow-apis)), the Flow API includes: * A file upload button when the request body content type is a binary file, as shown in the following image ![File upload button in the testing pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_upload_file.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=95f8661dfd607c6d8546803c6a372158) * A file download button when the response content type is a binary file > [!NOTE] > > If you need the API to return Base64-encoded text instead of a binary file, you need to use a [Script](../connectors/script) connector to encode the output as Base64 and set the response content type to something like `text/plain`. ### Invoke Access Users To see a list of users who can invoke a Flow API, open the **Settings** pane and click **Invoke Access Users**. This tab displays the list of all standard and service users who can invoke the Flow API, based on the [policies](../getting-started/administration/settings/user-roles#creating-roles-and-policies) assigned to them. Click **Add User** to grant access to other users who can invoke the Flow API. ![List of all users who can invoke the Flow API](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_invoke_access_users.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=c4d2937674ac97c184b6c76387040e6c) ### Unsupported Connectors Connectors in a Flow API must be connected in a way that results in one output message for each input message. Connectors that process files based on schedules or external processes are not supported. As a result, the following connectors are not allowed in Flow APIs: * [API](../connectors/api/api) * [Copy](../connectors/copy) * [Email Receive](../connectors/email-receive) * [Form](../connectors/form) * [FTP Server](../connectors/ftp-server) * [RSS](../connectors/rss) * [Schedule](../connectors/schedule) * [SFTP Server](../connectors/sftp-server) * [Webhook](../connectors/webhook) * [Workspace Receive](../connectors/workspace-receive) ### Example Flow API The example below shows a real use case for a POST-configured Flow API: ![Example Flow API in the workspace](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_example_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=3f4aa03f9e40bdf13fc2bc811308edf1) This Flow API performs the following workflow: 1. Data is received in the form of an encrypted X12 file at the Start node. 2. The OpenPGP\_Decrypt connector decrypts the X12 file and passes it to the X12 connector. 3. The X12 connector translates the X12 file to XML and passes the translated XML file to the JSON connector. 4. The JSON connector converts the XML file to JSON format and passes the converted file to the OpenPGP\_Encrypt connector, which encrypts the JSON file. 5. The encrypted JSON file is returned through the End node. In this example, each connector is linked to two other connectors—one before and one after. The Start and End nodes are automatically added to the endpoints of the flow. ## Testing Flow APIs After you create a Flow API, you can run sample requests to test your configuration. To open the Flow API testing pane, click the triangle-shaped icon in the header of the Flow API. ![Opening the Flow API testing pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_opentest.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=eeb88559edc4c9f9809d49d8c2261ea4) In the testing pane, the box on the left acts as the request, and the box on the right displays the response. Each box displays the expected data format chosen in the settings pane. ![Flow API testing pane](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flow_api_test_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=7691d9c31b57fba749b56d0f689e760a) To test your Flow API, enter a test query in the box on the left and click **Execute**. If the response matches the expected result, your Flow API is ready to use. If the response is incorrect, check the configuration of your Flow API and the configuration of each connector in the Flow API. ## Using the Flow API with an External Application The following steps explain how to invoke the Flow API from an external client. This example shows how to execute a Flow API from Postman, but you can use any client that can send REST requests. 1. Select the flow that you want to expose as an API by using the **Select Multiple Connectors** option at the bottom right of the flow designer. Then click **Create API Settings**. This action opens the **Create API Settings** dialog. 2. Select a method (for example, **POST**) as shown below, and give your API a meaningful name. ![Select a method in Create API Settings](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/arc_flow_method.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=97342be23558f2fddc588c0db115b2c0) 3. Click **Create API** to create the Flow API. 4. In the **Settings** pane, configure the Flow API to define the expected Content-Type value in the **Body** section and the Content-Type value that you want for the response in the **Response** section. As an option, you can provide sample data for the request and response. 5. Copy the path that appears in the **Request** pane. 6. Open the Postman application. Ensure that the same content type (**POST** in this case) appears in the Content-Type drop-down list. Then paste the path that you copied in Arc (step 5) into the request URL field. Specify the body of the request as the content that should start your flow. ![Copy and paste the Content-Type path in Postman](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/arc_body_content.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=10cbb6aa77d0242d127f60c99f43b835) 7. Authenticate to the Flow API. The Flow API uses the same authentication rules as the Admin API. This example uses the auth token header (**x-cdata-authtoken**) as the method for authenticating. To use that method: 1. Click the **Settings** gear icon on the navbar. Then, click **Add** to open the **Add User** dialog or to edit an existing user. 2. In the resulting dialog, select the check box **API Access** to display an auth token. (Copy this token to a safe place because the dialog is not shown again. If you lose or delete your auth token, you must create a new one.) Then click **Save Changes**. 3. Return to the Postman application and add the authentication credentials for the Flow API, as follows: 1. Click the **Headers** tab. 2. Paste **x-cdata-authtoken** in the **Key** field. 3. Paste the auth token that you copied in Arc into the **Value** field. 8. Click the **Send** button once your request is ready. You should see a response to the request that you sent, as shown in the example below: ![Response to the request in Postman](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/arc_response_to_request.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=716fc546cc6926d8f285f44e556e898d) ### Cross-Origin Resource Sharing (CORS) You can configure cross-origin resource sharing (CORS) for a Flow API by navigating to the [Admin API](../admin-api) portion of the [Security tab](../getting-started/administration/settings/security). CORS allows browser-based clients to connect to Arc. Without CORS, browser-based scripts cannot connect to Arc because of the same-origin policy enforced by the browser. This policy restricts client-side scripts and documents from loading resources outside their origin. The origin of a script consists of the protocol, host, and port.
SettingDescription
**Enable cross-origin resource sharing (CORS)** Whether or not CORS is enabled. The rest of these options are only available when you check this box.
**Allow all domains without '\*'** If enabled, domain origins are not restricted to a specific list.
**Access-Control-Allow-Origin** The comma-delimited list of domain origins to allow. Included as an HTTP response header.
**Access-Control-Allow-Credentials** Whether or not user credentials such as cookies are allowed in cross-origin requests. Included as an HTTP response header.
**Access-Control-Allow-Methods** The comma-delimited list of methods to allow. Included as an HTTP response header.
**Access-Control-Allow-Headers** The comma-delimited list of headers to allow. Included as an HTTP response header.
**Access-Control-Max-Age** The maximum duration (in seconds) that Access-Control response header values can be cached.
# Flows Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/flows Overview of the CData Arc Flows page, workspace grid, and flow canvas. The flows page enables you to create data-processing workflows by linking connectors together in logical sequences. Click **Flows** on the navbar to open the workspaces grid. It is composed of [workspace cards](./workspaces#workspace-cards) for each workspace you have access to. To work in a specific workspace, click the card to open the flow canvas for that workspace. ![Workspace grid showing multiple workspaces and options](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_grid.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=f18f1bd4e1d7e50d1a3212ac946a8ccd) > [!NOTE] > > You can toggle between two types of connector displays: > > * **Workspace View** displays the workspace cards that enable you to design your flows. The previous image is of the workspace view. > * **Connector View** displays a list of all connectors in your application. Use this view to perform bulk edits on connectors. See [Connector View and Bulk Editing Connectors](../connectors/bulk-edit-connectors) for details. ### Flow Canvas Example The following image shows a simple flow canvas that links a trigger **AS2\_Amazon** connector to a terminal **File\_Dropoff** connector (see [Connector Categories](../connectors#categories) for more information about trigger and terminal connectors): ![Flow canvas example](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_flow.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=623db5d096067bfa100412b4bc150945) With this connection in place, any files that the **AS2\_Amazon** connector processes are automatically passed to the **File\_Dropoff** connector as the final step of the flow. The File connector drops off files to the underlying disk. See [Designing a Flow](./designing-a-flow) for more information about how to link connectors in a flow. # Workspaces Source: https://docs.arc.cdata.com/26.4/self-hosted/en/flows/workspaces Manage CData Arc workspaces to isolate flows, control user access, and organize connectors. **Workspaces** enable you to isolate flow canvases. Connectors and flows that are configured in separate workspaces are held in entirely separate folders on disk, which allows a complete separation of processes for organizational purposes. Only the connectors from one workspace are displayed at a time, so using multiple workspaces can help reduce clutter on the **Flows** page. For example, a single flow canvas could contain an automated trading process for each of Amazon, Target, and Walmart but these independent flows could also be separated out into new workspaces, as shown in the following image: ![Three workspace cards for Amazon, Target, and Walmart](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/Workspaces.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=ba400c1a80920b263e56fb38b8ad2c27) Using workspaces to compartmentalize flows also provides a means to easily control the permissions of users and their access to those flows. Using the previous image, the user account assigned to Jenny might have full access to the Amazon workspace, more limited access to the Target workspace, and no access at all to the Walmart workspace. For more information about permissions, see [User Roles](../getting-started/administration/settings/user-roles). Connectors in one workspace cannot be connected to connectors from a different workspace. The [Activity page](../getting-started/administration/activity) collects and displays data across all workspaces. You can filter by workspace name on the **Messages** tab, but workspaces are treated as functionally the same for application monitoring and management. ## Using the Workspaces Grid When you click the **Flows** tab, you are presented with the workspaces grid. From here you can perform actions on individual workspaces or use the checkboxes on the [workspace cards](#workspace-cards) to select multiple workspaces on which to perform bulk actions. ![Workspace grid showing multiple workspaces and options](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_grid.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=f18f1bd4e1d7e50d1a3212ac946a8ccd) You can perform the following actions: * Click ![Refresh icon](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_grid_refresh.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=a12af31e69f6781217c93eb9c65eb814) to refresh the selected workspaces (all workspaces if none are selected). * Click ![Help icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_help.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=5490fe32dad5fde973ad68ffd6ef2e13) to open this documentation in a new browser tab. * Click **Add Workspace** to open a dropdown with the following options: * **Add New Workspace**: Creates a new workspace. * **Import Workspace**: Opens the **Import** dialog. Navigate to the `.arc` file you want to import and choose **Import**. Then, configure the import options that you want and click **Import** to check for conflicts and to complete the import. See [Setting Import Options](../getting-started/administration/importing-exporting-flows#setting-import-options) and [Resolving Import Conflicts](../getting-started/administration/importing-exporting-flows#resolving-import-conflicts) for details. * Click **Export Workspace** to export the selected workspaces into a `.arc` file (only available when at least one workspace card is selected). * Click **Remove** to delete the selected workspaces (only available when at least one workspace card is selected). * Click **Deselect** to clear the selected workspace cards (only available when at least one workspace card is selected). The search bar lets you search for specific workspaces, [connectors](../connectors), connector types, or [Flow APIs](./flow-api). ![Workspace search results](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_searchresults.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=4889874828f5935e399b6227758a704c) Clicking on a result takes you to that object in its workspace card. Use the **Sort** dropdown to sort the workspace cards by the following criteria. Click **Reset** to return to the default sort order. * Alphabetical order * Last modified date * Last accessed date ## Workspace Cards Each workspace card is a snapshot view that shows the workspace name along with a count of the connectors, Flow APIs, and users with access to that workspace. ![Workspace card](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_card.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=82385189d96aeac6fe1554e5aa67164d)
IconDescription
![Chain icon](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_card_chain_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=0a5ea93103698271c89db224d7cd37ec) Hover over this icon to see the list of connectors in the workspace.
![Screen icon](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_card_screen_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=97c0465928eedcc86b67bbb9ad2d5667) Hover over this icon to see the list of all Flow APIs in the workspace and the HTTP method being used.
![User icon](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_card_user_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=ec7a0d179607a21fac80162474bce1a3) Hover over this icon to see the list of users with permissions on the workspace.
Hover over a card and click the ellipsis to see additional options: ![Workspace card ellipsis menu showing Import Flows, Export Flows, Settings, and Remove options](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_card_additional_options.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=6d6d11761e19d7044d938e8516bfd2c5) * **Import Flows**: Opens the **Workspace Import** dialog. Navigate to the `.arcflow` file you want to import and choose **Open**. Click **Next** to set your import options, then click **Next** to check for conflicts and to complete the import. See [Setting Import Options](../getting-started/administration/importing-exporting-flows#setting-import-options) and [Resolving Import Conflicts](../getting-started/administration/importing-exporting-flows#resolving-import-conflicts) for details. * **Export Flows**: Opens the **Workspace Export** dialog, which allows you to export the workspace contents to a `.arcflow` file. See [Exporting a Workspace as a `.arcflow` File](../getting-started/administration/importing-exporting-flows#exporting-a-workspace-as-a-arcflow-file) for details. * **Settings**: Opens the **Workspace Settings** panel. * **Remove**: Deletes the current workspace. > [!TIP] > > To import a global `.arc` file, use the **Add Workspace** dropdown on the **Flows** page and choose **Import Workspace**. ## Default Workspace Arc includes a special *Default* workspace that you cannot delete. You are not required to use multiple workspaces because you can configure all flows in the default workspace. The default workspace stores all of the application data files and configuration files in a `data` folder in the Application Directory. The data and configuration files for any other workspaces are held in a `workspaces` folder in the Application Directory. ## Navigating and Managing Workspaces from a Flow Canvas From the workspaces grid, click into a workspace. You can use the top toolbar of a flow canvas to manage your workspaces: ![Flow canvas workspace toolbar options](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_canvas_workspace_options.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=a31bc18b24a8bd884b6fe0af951369b7) * Use the workspace selector dropdown on the left to switch to a different workspace. * ![View users icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_view_users.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=62e4398c95b975a979ab670c312bfce6) See which users can access this workspace, or add new users. This option is only visible to Arc admin users. * Click the ellipses for additional options, then make your selection: * Select **Import Flows** to import a `.arcflow` file that was exported from another Arc instance. Navigate to the `.arcflow` file you want to import and choose **Open**. Click **Next** to set your import options, then click **Next** to check for conflicts and to complete the import. See [Setting Import Options](../getting-started/administration/importing-exporting-flows#setting-import-options) and [Resolving Import Conflicts](../getting-started/administration/importing-exporting-flows#resolving-import-conflicts) for details. * Select **Export Flows** to export the workspace contents to a `.arcflow` file that can be imported into another Arc instance. Set your export preferences, then click **Export** to create the file. See [Setting Export Options](../getting-started/administration/importing-exporting-flows#setting-export-options) for details. * Click **Settings** to open the workspace-specific [Settings](#workspace-settings) panel. * ![Help icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_help.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=5490fe32dad5fde973ad68ffd6ef2e13) Click the **Help** button to open the documentation for the flow designer in a new browser tab. * Use the search bar to search for connectors, [flow APIs](./flow-api), saved views, and [notes](./designing-a-flow#using-notes) in the current workspace. When you find what you are searching for, click it to be taken to the object in the canvas. * ![Cursor select toggle](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_cursor_select.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=3e1301bd030ee0a3154208601ea67d03) Use the toggle to set what action the cursor performs. The options are: * **Move**: Allows users to click and drag around the canvas to navigate within the canvas. This is the default action. * **Multi-Select**: Allows users to draw a rectangle around a group of connectors. This is useful when creating flow APIs or when you need to move or delete multiple connectors at once. Right-click on the rectangle to see the available actions you can take on all the objects in the rectangle. * ![Undo icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_undo.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=cd3114a7e05488e2ab6c5a0749934def) Undo your most recent action. * ![Redo icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_redo.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=a7782ce5b98e3485924543a51b4fb08c) Restore a previously undone layout on the canvas. * ![Save view icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_save_view.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=65c891ecc87df1583bd03cd9d8935ce7) Save the current workspace view. * ![Auto-format icon](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flows_autoformat.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=f8d448ac96f4ba28a116cb9a72fbda4f) Auto-format and lay out the entire flow canvas. If you select multiple connectors in the workspace, use this to auto-format just the selected connectors. > [!TIP] > > Click the **Flows** breadcrumb link to return to the workspace grid. > > The small map in the bottom right corner of the flow canvas shows silhouettes of the connectors in the workspace. The current view of the canvas is represented by the white rectangle in the map. As you move around the canvas, the white rectangle also moves, so you always have a high level picture of where you are in the workspace. ## Workspace Settings Workspace Settings are a set of tabs that enable you to control specific settings on a per-workspace basis. To open, hover over a workspace card and click **Settings** in the ellipsis menu, or open a workspace and click **Settings** in the toolbar ellipsis menu. Use the tabs on the right side of the panel to switch between categories. ![Workspace settings panel](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/workspace_settings.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d5fb3d0f5845ee3563b7d587e6e9837b) For the Alerts, Performance, and Cleanup categories, you must use the **Override Global Settings** toggle for the workspace-specific settings to take effect. These settings are a subset of the options on the [Settings page](../getting-started/administration/settings/settings-page). You can configure: * [Alerts](../getting-started/administration/settings/alerts) * [Performance Settings](../getting-started/administration/settings/performance-settings) * [Cleanup Settings](../getting-started/administration/settings/cleanup-settings) * [Git Version Control](../getting-started/administration/git-version-control): Use the **Enable Git Version Control** toggle to turn on Git version control for the workspace (not available in the [Default](#default-workspace) workspace). * Miscellaneous: Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. ### Allowing ArcScript in the Alert Subject On the **Alerts** tab, select **Allow ArcScript in Subject** to use [ArcScript](../scripting) in the alert email subject. When this setting is enabled, a `` button appears in the **Subject** field. Click the button to open the ArcScript editor. The **Allow ArcScript in Subject** setting is available at the application, workspace, and connector levels. If the setting is configured at both the workspace and connector levels, the connector setting overrides the workspace setting. # Two-Factor Authentication Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/2fa How to enable and configure two-factor authentication (2FA) in CData Arc. Enabling Two-Factor Authentication (2FA) in CData Arc adds an extra layer of protection to your Arc instance. 2FA enhances security by requiring two forms of verification when you log in: * Something you know: Your password. * Something you have: A temporary code from a supported authenticator app. If your password is ever compromised, 2FA helps prevent unauthorized access by ensuring that only the real user can complete the login process. Arc supports 2FA through the use of authenticator apps such as Google Authenticator, Microsoft Authenticator, and Authy. To configure 2FA in Arc, follow these steps: 1. Click your user avatar in the lower left corner of the navbar to open your user details. 2. On the **User Details** page, click the **Set up** link in **Two-Factor Authentication**. 3. Re-enter your password to verify your account. 4. After you verify your account, a setup window opens asking you to download your preferred authenticator app and scan the QR code in the window. ![2FA QR code setup window](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/2fa_setup.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=d812c4e33356aae1c771eebdbdada0e2) 5. After you scan the QR code with the authenticator app, CData Arc is added to your authenticator. 6. Enter the **Security Code** from your authenticator app, then click **Next**. If authentication is successful, a success message appears. Once 2FA is configured, anytime you log into the application, enter your password as normal. You are then presented with another prompt asking for the security code from your authenticator app. > [!NOTE] > > * You must set up your own 2FA. > * Users with an [Admin role](./settings/user-roles#admin-role) can remove 2FA from other users by navigating to **Users > \[user name] > Two-Factor Authentication > Remove**. > * You can use the same method to remove 2FA from your own account. > * You can check to see if you have set up 2FA for your account by clicking on your avatar in the left navbar. # Activity Page Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/activity How to use the CData Arc Activity page to view and work with messages, SLAs, and application logs. CData Arc maintains logs for multiple types of activities, including [messages](#messages-tab), [SLAs](#slas-tab), [application](#application-logs) and [access](#access-logs) requests, [automation](#automation-logs) decisions, [audit](#audit-logs) configuration changes, and [profile](#profile-logs) activities. Use the **Activity** page to view and work with your logs. Each log type is stored as a separate table in the application database. View the application, automation, access, audit, and profile logs from the [Logs](#logs-tab) tab. > [!NOTE] > > Date/Time entries are always localized to the local time zone and date format settings of your browser, so even when you browse to a server hosted in a different time zone, the displayed timestamps reflect the local time. ## Searching and Filtering the Logs The top of each **Activity** page tab has a search bar that you can use to search all the logs on that page. Click the funnel icon to open a context-sensitive filter section with different options for each log type. For example, as shown in the following image, the Messages tab has filters for Tracked Headers, time, connector type, workspace, connector Id, and status. ![Log filter panel](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/log_filter.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=3e79ce7c511b9dcf55d8569f44695693) You can use filter options *and* the search bar in an individual search to narrow the results that are returned. The image below shows messages in the last 24 hours with a status of Success and the word 'REST' in the message text. ![Filtered messages example](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/log_filter_messages.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=06a52dd69155d4a12109cccad6abebaa) ### Tracked Headers Arc supports using additional searchable metadata (Tracked Headers) as an additional way to filter results on the **Messages** tab. You can add tracked headers in a number of ways: * The **Tracked Headers** field in the **Advanced Settings** section of the **Settings > Advanced** page lets you provide a comma-delimited list of custom header names. * Some connectors, like X12, have settings in the Advanced tab that let you to opt-in to the tracking of certain headers, like ISA08. * The XML Map connector lets you right-click an element and add it as a tracked header on the output message. * The `_message` item in ArcScript lets you set tracked headers through a `_message.trackedheader:HeaderName` syntax. To see other examples of using `_message`, read [CData Arc Message](../../scripting/introduction-to-arcscript#message-_message). Arc adds metadata headers to files as they are processed in the flow. Find the headers by hovering over a message and clicking **View Details**. You can do this by: * Expanding a message in the **Messages** tab * Opening the **Message Details** page * Viewing the Transactions tab for a connector ## Messages Tab The **Messages** tab provides insight into a message's entire journey through a flow, which can contain many transactions. You might often want to see what happened with a message that you received, and this lets you see the full picture of the message as it traveled through the flow. The Messages table contains the full list of messages. By default, the list is sorted by the **Latest Message** column, but you can click any column name to sort by it. It displays the following details: * **Latest Message**—the latest filename for the message with a timestamp for the last activity related to this message. Click this link to open a window to preview the message content and optionally download, copy, or format it. * **Flow Start and Last Processed**—the [Connector Id](../../connectors) where the message originated, and the Connector Id of the last connector to process a transaction related to the message. Click a Connector Id to open the connector's configuration page in the [flow designer](../../flows/designing-a-flow). * **Status**—the status of the message. The options are: * Success * Warning * Error * Pending * Skipped * **Total Processing Time**—the amount of time that the message was in the flow for processing The search bar and filters section let you filter based on the columns mentioned above, as well as any Tracked Headers. The following image shows an example filter: ![Messages filter example](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/messages_filter.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=b0721a45b78ccda0c2d2fdc466bb790c) To open the **Message Workflow** page for a message, click the arrow that appears on the far right when you hover over the message. This page displays all steps in the flow, along with the associated documents, other details, and the logs. ![Message workflow page](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/message_view_details.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=9b1744ec414f2493c2953a61fb41294f) Use the expander to the left of a message to see the message **Details**, as shown below. ![Message details card](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/message_details.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=c2f9fadacce852ca1828f0a10410ff61) The card contains: * The Message Id * The time at which the message originated * A list of Tracked Headers associated with the message * A list of Other Headers associated with the message (not tracked) * Messages that pertain to the message status. For example, if the status is *Error*, the card shows the error message that caused this status in the summary. The **Logs** tab for the message lists logs that pertain to the processing of the message. To preview the log contents in the UI (and optionally download the file), click the link in the **File Name** column. To view or download a log entry, click the **Logs** tab. Hover over the entry you are interested in, then use the download icon ![download icon](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/download_icon.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=2608c3414af9587ebd7421fdc11859e0) to download an individual log file. Alternatively, use the **Download All Logs** button to download a zip file containing all logs that pertain to the message. ## Diagnosing Transaction Errors When a transaction fails, you can use the AI model configured for Arc to analyze the failure and suggest a cause and next steps. The analysis runs only when you start it. Arc never diagnoses a transaction automatically. > [!NOTE] > > Diagnosing errors requires an AI provider. Before you start, configure one in the [AI Settings](./settings/ai-settings) section of the **Settings > Advanced** page. ### Starting a Diagnosis A **Diagnose error** link appears below the error message of any failed transaction. You can find it on the **Message Workflow** page for a failed message (open it from the **Messages** tab), in the message **Details** card when you expand a failed message on the **Messages** tab, and on a connector's **Transactions** tab. ![Failed transaction with the Diagnose error link](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/diagnose-error-activity-page.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=26c9c78b8cf39c73793d261d90737f44) 1. Click the **Diagnose error** link to open the **Error Diagnosis** window. The window shows the error message on the left. ![Error Diagnosis window before a diagnosis is run](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/diagnose-error-window.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=abada6eed098e8f037bd02184e53d205) 2. Click the **Diagnose Error** button to start the analysis. ### Reading the Diagnosis The diagnosis appears on the right side of the window, organized into the following sections (not every section appears on every diagnosis): | Section | Description | | - | - | | **Summary** | A short description of what failed. | | **Verdict** | The type of problem (for example, a configuration problem) and how confident the analysis is in that conclusion. | | **Root Cause** | The explanation of why the error occurred, based on the evidence gathered. | | **Remediation** | Suggested steps to resolve the error. | | **More Information Needed** | Details the analysis could not determine, which you might need to check yourself. | | **Evidence** | The log entries and settings that support the conclusion. | | **Other** | Any findings that do not fit the sections above. | ![Diagnosis window showing the Summary, Verdict, and Root Cause sections](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/diagnose-error-window-result.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=5d6dbab12127719f846afb7a32ac6cd4) Individual findings can include the following labels: **Confidence**, **Risk**, **Severity**, and **Type**. A finding can also include a **Documentation** link to a related help topic. Click **Copy** to copy the diagnosis to your clipboard. If the analysis cannot be completed, the window displays **Unable to Diagnose**. Click **Retry** to run the analysis again. ### Data Used for the Diagnosis Arc sends information about the failed transaction to the AI provider you configured in AI Settings. It does not send the message payload or the message headers. The information includes: * The transaction record and its detail logs * The connector's settings, flow, certificates, and recent activity * The related request, application, and service logs * The identity and trading partner configuration for the following connector types: AS2, AS3, AS4, OFTP, GISB, RNIF, PIP, PEPPOL, X12, EDIFACT, TRADACOMS, VDA, SFTP Server, FTP Server, TCP Server, and OpenPGP Arc also includes excerpts of the Request, Response, and Headers logs, filtered by keyword to the portions relevant to the error. The Headers log contains the HTTP headers of the request and response that created the transaction. Content that is not UTF-8 text is excluded. When Arc excludes or truncates information, it notes this in the information sent to the model. > [!NOTE] > > Diagnosing an error is read-only. Arc does not apply fixes, roll back changes, or modify any configuration as a result of a diagnosis. > > AI-generated diagnoses can be incorrect, so verify the suggested cause and remediation steps before you change your configuration. Review your AI provider's data handling terms to confirm that sending this information to the provider meets your organization's requirements. ## SLAs Tab The SLAs tab stores information on every SLA configured on individual [connectors](../../connectors). It includes information on which [workspace](../../flows/workspaces) the SLA is associated with, as well as status information and the time remaining in that status. It displays the following details: * **Connector Id**—the Id of each connector with a defined SLA * **Workspace**—the name of the associated workspace * **Previous Status**—the prior SLA status (for example, *OK*) * **Current Status**—the current SLA status (for example, *At Risk*) and the time remaining before the SLA changes status again (for example, to *In Violation*) ## Logs Tab The **Logs** tab contains five sets of logs: Application, Automation, Access, Audit, and Profile. By default, each table includes a summary count of today's logs, and a total count of all logs. The application logs also group similar entries into patterns. Click the **See All** link in each section to populate the window with the full list of logs in that category. From here you can search, filter, and download logs. > [!NOTE] > > Logs maintenance is managed by using the **Interval** value in the [Cleanup Settings](./settings/cleanup-settings) section of the **Settings > Advanced** page. The [automation logs](#automation-logs) also have their own retention rules: see [Retention](#retention). ### Application Logs The application logs record any application-level errors that occur during processing and any requests made to application resources. The error messages here can help provide context to errors for specific transactions. Use this tab to verify that external trading partners' web requests are successfully routed to a connector within the Arc flow. When the application receives a web request at a public endpoint, it attempts to route the message to a specific connector based on message headers (for example, AS2 Identifiers). If an error prevents the application from routing the message successfully, this error appears on the **Application** tab. To help you spot recurring issues and noisy activity at a glance, Arc groups similar application log entries into *patterns*, as shown in the following image. A pattern represents a set of messages that share the same structure, with the variable parts of each message (such as IDs, counts, and task names) replaced by a `<*>` wildcard. For example, the messages `Stopped task [CheckIntegrityTask]` and `Stopped task [PortHealthCheckTask]` are both grouped under the pattern `Stopped task <*>`. ![Application Log Showing Patterns](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/application_log_patterns.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=61739f2d5ad7ba4bce1bdd09e78a85ea) By default, the **Application Logs** card on the **Logs** tab shows this condensed, pattern-based view. At the top of the card, the **Error** and **Warning** tiles show the current counts for those levels. Below the tiles, a table lists each detected pattern, and each pattern row displays the following details: * **Count**—the number of log entries that match the pattern * **Trend (Time Period)**—a sparkline showing how often the pattern has occurred over time * **Level**—the log level of the pattern (for example, *Info*) * **Pattern Template**—the shared message structure, with variable values shown as `<*>` * **First Seen**—the timestamp of the earliest entry that matches the pattern The bottom of the card shows how many logs were recorded today and the total number of logs. To see additional details on the pattern groups, click **See All**. The full log window opens in the **Patterns** view by default; hover over any row in the table and click **View Details** to see details on the logs which make up that pattern. To view the individual log entries instead of the grouped patterns, click the **Logs** button in the toolbar. In the **Logs** view, the Application table includes a separate row for each application event. It displays the following details: * **Date/Time**—the timestamp for the event * **Level**—the log level. Review the information in the **Message** column for details. The options are: * None—events are not logged * Error—information about errors encountered during processing * Warning—information about warnings encountered during processing * Info—general information about the processing, including any errors and warnings (if applicable) * Debug—detailed debugging information for both successful and failed processes * Trace—detailed trace information for both successful and failed processes (useful if you need to send logs to CData Support) You can change the log level in the **Advanced Settings** section of the **Settings > Advanced** page. However, increasing the verbosity of the logs can affect database performance, especially for less performant database types like SQLite or H2. Only adjust the log level if you have a specific reason to do so. > [!NOTE] > > Logging levels **Debug** and **Trace** might log sensitive information, including file information and SSL certificates. Although sensitive details such as passwords are masked, you should review these logs for sensitive information before sharing them outside of your organization. * **Resource**—the resource that produced the message * **Message**—the message text * **Instance Id**—the machine on which the event occurred ### Automation Logs The automation logs record what the automation service decided for each connector: whether the connector was due to run, whether it was triggered, why it was skipped, and whether a run succeeded, failed, or completed without receiving any message. Use these logs to confirm that a scheduled connector actually polled, even when the poll produced no messages and no errors. Automation logs are separate from the [application logs](#application-logs), so these entries do not appear in the application log. Both the **Automation** section and the full page display the following details: * **Date/Time**—the timestamp for the entry * **Level**—the severity of the entry: Debug, Info, or Error * **Connector**—the connector and workspace the entry applies to. Click the connector name to open its configuration page. * **Action**—the automation action the entry covers: Receive, Scheduled Send, Check Pending, or Service (a service-level decision that applies to the connector as a whole rather than to one action) * **Decision**—what the automation service decided (see the [Decisions](#decisions) table for details) * **Next Run Time**—the next time the connector is scheduled to run, when the entry involves a schedule evaluation * **Reason**—a text description of the decision * **Instance Id**—the Arc instance that wrote the entry The **Automation** section on the **Logs** tab shows recent entries. Click **See All** to view the full set of logs. Here you can use the search bar and the filters section to narrow the list by any column. Use **Download Logs** to download a subset of logs. #### Decisions Each entry has one of the following decisions: | Decision | Meaning | Recorded at | | - | - | - | | **Success** | The run completed and produced messages. | Always | | **No Messages** | The run completed without producing messages. | Always | | **Failed** | The run failed. | Always | | **License Skipped** | A license error prevented processing. | Error, Info, and Debug | | **Not Started** | A due action did not start a worker. | Error, Info, and Debug | | **Triggered** | A due schedule started a worker. | Info and Debug | | **Backoff Skipped** | The connector is in [backoff](../../connectors/connector-backoff). | Info and Debug | | **Lock Skipped** | A connector lock prevented a due run from processing. | Info and Debug | | **Not Due** | The schedule was not due. | Debug | | **Already Handled** | Another instance handled the due schedule. | Debug | At the Debug level, entries are also recorded for service-level **Lock Skipped** decisions (the connector lock was unavailable when no run was due) and for **Triggered** decisions on the Check Pending action. **Triggered** is a reliable confirmation that a run started. The scheduler advances the schedule as soon as it finds a run due, so a due evaluation alone does not prove that anything happened. Arc writes **Triggered** only after the worker starts, and records every due action that does not start as **Lock Skipped** or **Not Started**. #### Log Level Automation entries follow each connector's own **Log Level** setting (on the connector's **Advanced** tab), not the application log level. You can get automation detail for a single connector without raising the log level for the whole application. Outcomes are always recorded, whatever the level. **Success**, **No Messages**, and **Failed** describe a run that executed, so they are written unconditionally. Raising the level does not reveal whether a run succeeded, because that is always recorded. It reveals why a run did not happen: * **Error** adds **License Skipped** and **Not Started**. * **Info** adds **Triggered**, **Backoff Skipped**, and **Lock Skipped** for a due run that was blocked. * **Debug** adds the evaluations behind each decision, including **Not Due** and **Already Handled**. At levels below Debug, Arc writes **Backoff Skipped** and **License Skipped** at most once every five minutes for each connector. At Debug, every skipped evaluation is recorded. #### Retention The daily cleanup removes automation log entries that are older than seven days, or older than the **Interval** value in [Cleanup Settings](./settings/cleanup-settings) if that is shorter. Arc also limits the table to about 100,000 entries. Because the daily cleanup enforces that limit, the table can briefly exceed it. ### Access Logs The access logs record entries for requests that are made to endpoints that it provides. This includes API requests, connectors that expose a public endpoint (for example, Webhook), and public endpoints related to receiving data (for example, public.rst). It displays the following details: * **Date/Time**—the timestamp for the transaction * **Method**—the HTTP method used * **URL**—the URL * **User**—the authenticated user who made the request. If the user who made the request was anonymous, this entry is blank. * **User Type**—the account type of the user who made the request: **Standard**, **SSO**, **Active Directory**, **LDAP**, or **Service**. Each type is shown with a distinct icon. See [User Types](./settings/users#user-types) for more information. * **Remote IP**—the IP Address where the request originated * **Instance Id**—the Id of the instance that served the request ### Audit Logs The audit logs store each change made to the application configuration. This includes all actions, such as creating connectors, configuring connectors, updating profile settings, and so on. It displays the following details: * **Date/Time**: the timestamp for the configuration change * **Action**: the type of change that was made. The options are: * GET * POST * PUT * MERGE * PATCH * **Resource**: either the Admin Console (the web UI that serves as the typical interface for managing and configuring the application) or the [Admin API](../../admin-api) (the REST-based API that can be consumed to manage and configure the application). * **Message**: detailed text description of the change, including which resource was affected, the specific field or setting that was modified, and the previous and new values * **User**: the user who made the change * **User Type**: the account type of the user who made the change: **Standard**, **SSO**, **Active Directory**, **LDAP**, or **Service**. Each type is shown with a distinct icon. See [User Types](./settings/users#user-types) for more information. You can query the Audit logs to perform audits of the application configuration. ### Profile Logs The profile logs record activity from configured communication [profiles](./profiles). This includes file transfers, protocol-level events, and server activity for profiles such as AS2, AS4, and SFTP Server. It displays the following details: * **Date/Time**—the timestamp of the activity * **Connector**—the Id of the connector involved * **File**—a link to open the log file for the profile * **Status**—the status of the action. Options are Complete, In Progress ## Tamper-Evident Logging Arc includes cryptographic tamper-evident logging on all [Audit logs](#audit-logs). This ensures that the user is alerted anytime that log data is altered or tampered with. It works by creating an immutable audit trail of system activities by cryptographically linking log entries in a chain, making unauthorized modifications immediately detectable. Each log entry includes a cryptographic hash of both its own content and the previous entry's hash, creating a verifiable chain where any alteration to historical records breaks the chain's integrity. This approach helps organizations comply with regulatory requirements such as HIPAA and GDPR. Integrity checks run hourly, as part of the application health checks. If tampering is detected, an error is logged. If email alerts are configured on the [Alerts](./settings/alerts) tab, an email is sent to administrators. Administrators can use the [integrityResetTampering](../../scripting/operations/op-integrity-reset-tampering) operation to reset the hash chain and clear tampering warnings. ## Debug Logging Some connectors have settings in the Advanced configuration tab to enhance the verbosity of transaction logs. These detailed logs are useful when debugging, and are often necessary when requesting support from *[arcsupport@cdata.com](mailto:arcsupport@cdata.com)*. The most common Advanced configuration setting is **Log Level**. When set to *Debug*, additional information is added to the transaction logs that are already generated when processing a transaction. On connectors that run on a schedule, **Log Level** also controls how much automation activity is recorded in the [automation logs](#automation-logs). Some connectors that send outgoing requests also have a **Log Requests** setting. When set to *True*, transaction logs include a copy of the request as it was sent to the remote server. This can be useful in narrowing the scope of any issues receiving server responses. Some connectors that issue queries against remote systems have **Verbosity** and/or **Logfile** settings. The **Verbosity** should be set between 1 and 5 (where 5 is the most verbose, and recommended in most cases). When **Logfile** is an available setting, set it to a path on disk to write the dedicated logfile; you can provide this file to Support with the transaction logs generated when processing a transaction. # Administration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/administration Overview of CData Arc administration topics including the Dashboard, Profiles, Settings, Activity, and Security Best Practices. CData Arc is a web application, and can be hosted using the embedded web server included with the installation or on an external web server. The following topics contain information related to administering the application. ## Dashboard Page The [Dashboard](./dashboard) topic describes how to configure your landing page to display metrics and other dashboard data that is important to you and your organization. ## Profiles Page The [Profiles](./profiles) topic introduces the connectors that require partner connection settings that are used globally across the application. ## Settings Page The topics in the [Settings](./settings/settings-page) section cover the global Arc settings, including: * User Management and Roles * AI settings * Certificates * Alerts * Admin API * Connections * Single Sign-On (SSO) * Vault * Advanced settings * Licensing ## Activity Page The [Activity](./activity) topic covers two distinct features. It details the various types of logs in Arc, including: * Messages * Transactions * Application * Access * Audit It is also where you find details about the Service Level Agreements (SLAs) configured across the application, and their statuses. ## Importing and Exporting The [Importing and Exporting](./importing-exporting-flows) topic explains how to import or export connectors, flows, or workspaces. It includes details on how to configure import and export tasks, and how to resolve potential import conflicts. ## Reports The [Reports](./reports) topic describes how to create, schedule, and manage reports for various data trends in the application. ## Git Version Control The [Git Version Control](./git-version-control) topic covers how to use Git-based version control to track configuration changes with full audit trails. ## Security Best Practices The [Security Best Practices](./security-best-practices) topic provides information on security-related features available in Arc and describes the best practices we recommend you follow. # Clustering Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/clustering How to configure multiple CData Arc installations to work together for high availability and scalability. Clustering allows multiple CData Arc installations to work together, processing the same data with the same configuration. Workloads can be distributed horizontally across clustered Arc installations to improve scalability and ensure availability. ## Overview To take advantage of the high-availability and failover features supported in Arc, the application should be installed on multiple systems in the same server farm (the same cluster). A load balancer then distributes incoming traffic across the multiple systems hosting Arc instances. ![Cluster behind a load balancer](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/clustering_load_balancer.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=a0e8b506021f12510b445b92acac559e) When configured for clustering, each Arc installation in the server farm uses the same application configuration, processes data to and from the same locations on disk, and logs transactions in the same database tables. As a result, multiple instances of the application behave like a single instance, and any particular instance can go down without jeopardizing the performance of the cluster. ## Configuring Arc for Clustering After Arc has been installed on each node in the cluster, each installation should be configured to use the same *Application Database* and *Application Data Directory*. ### Licensing You must apply a unique license key to each node in the cluster. Go to the licensing page for each node and apply one of your keys. The application creates a license file in the shared directory of the cluster. The license file name includes the machine name of both nodes in the shared directory. ### Application Database Arc uses a database to log transaction history and any errors that occur in the application. Each instance of Arc should be configured to use the same application database to ensure that all files processed end up consolidated in the database. #### .NET Edition To configure the application database in the .NET edition, the `AppDb` environment variable must be set to include the appropriate connection string and provider. To accomplish this, modify the `Web.Config` file in the installation www folder. In this file is a commented-out XML element called *connectionStrings*, for example: ```xml theme={null} ``` Un-comment the *connectionStrings* element and set the `connectionString` and `providerName` attributes to the appropriate connection parameters for the desired database. If Arc can successfully establish a connection with this connection string, it uses this database as the application database. #### Embedded Java Server When using the Cross-Platform edition with the embedded Jetty server, the application database is configured in the `arc.xml` file located in the installation directory webapp folder. In this server configuration file, the `APP_DB` environment variable must be set to a JDBC connection string containing the appropriate connection parameters for the desired database. For example: ```xml theme={null} APP_DB jdbc:cdata:mysql:Server=MySQLServer;Port=3306;Database=mysql;User=user;Password=password ``` If Arc can successfully establish a connection with the `APP_DB` connection string, it uses that database as the application database. #### External Java Server When using the Cross-Platform edition with an external Java servlet (any server other than the Jetty server that is included with the application), the details of configuring the application database depend upon the specific servlet used. Using the syntax appropriate for the specific servlet, use one of the following approaches to configure the server: * Define a JNDI datasource to include the connection properties for the target database. * Set the `APP_DB` environment variable to a JDBC connection string. If Arc can use the JDNI datasource or `APP_DB` connection string to connect to a database, it uses that database as the application database. ### Application Data Directory Arc stores all configuration data and application data in a folder on disk called the data directory. When clustering, each instance of Arc should be configured to use the same data directory. This ensures that all instances are processing the same files and using the same configuration. #### .NET Edition To configure the application data directory in the .NET edition, the `AppDirectory` environment variable must be set to the path where the directory should be created. To accomplish this, modify the `Web.Config` file in the installation directory www folder. In this file is a commented-out XML element called *AppDirectory*, and below this an element called *appSettings* where a custom data directory location can be specified: ```xml theme={null} ``` Un-comment the *appSettings* element and set the `AppDirectory` key value to the appropriate path on disk for the data directory. If Arc can find the path, and has the appropriate permissions to read and write at the given path, it creates the data folder in the specified directory. #### Embedded Java Server When using the Cross-Platform edition with the embedded Jetty server, the application data directory is configured in the `arc.xml` file found in the installation directory webapp folder. In this server configuration file, the `AppDirectory` environment variable must be set to the path to the desired directory. The following example demonstrates what this might look like when setting the data directory to a shared folder on a mounted drive: ```xml theme={null} AppDirectory /mnt/shared/arc ``` If Arc can find the path, and has the appropriate permissions to read and write at the given path, it creates the data folder in the specified directory. #### External Java Server When using the Cross-Platform edition with an external Java servlet (any server other than the Jetty server that is included with the application), the details of configuring the application data directory depend upon the specific servlet used. Using the syntax appropriate for the specific servlet, the `AppDirectory` environment variable must be set to the path to the desired directory. If Arc can find the `AppDirectory` path, and has the appropriate permissions to read and write at the given path, it creates the data folder in the specified directory. ### Locking and Concurrency Arc uses locks to ensure that multiple instances do not interfere with each other or process the same file twice. In a clustered environment, efficient locking is critical to maintain throughput and prevent collisions. Thus it is strongly recommended not to cluster Arc instances across multiple server farms, such that file system latency might come into play. Setting a shared Application Directory for each instance of Arc is sufficient to ensure that the file locks are respected by each instance. ### Configuring Reverse Proxies A reverse proxy is a server that sits between clients and backend servers, intercepting incoming traffic and forwarding it to the appropriate destination. When configured properly, reverse proxies can preserve the real client IP address through header forwarding while protecting your backend infrastructure from direct exposure. This architecture provides benefits such as load balancing, improved security, caching, and SSL termination. When a reverse proxy forwards client information through headers like `X-Forwarded-For`, it is critical to restrict which proxies are trusted. If you don't explicitly trust only your known proxy IP addresses, malicious users can fabricate these headers to spoof their origin, potentially bypassing security controls or logging mechanisms. #### Cross-Platform Edition To configure reverse proxies in the Cross-Platform edition, set the [proxyMode](../installation/arc-properties-configuration#cdata-http-proxymode) setting in the `arc.properties` file to true, as shown in the following string: `cdata.http.proxyMode=true`. When `proxyMode=true`, the embedded Jetty server uses `ForwardedRequestCustomizer` to extract and apply the real client IP from forwarded headers. If you are using an external Java servlet container, configure request rewriting at the servlet level to handle forwarded headers appropriately. #### .NET Edition To configure reverse proxies in the .NET edition, you must provide a comma-separated list of header names that the app should use to determine the client IP address in the **HTTP Forwarded Header** field on the [Proxy Settings](./settings/proxy-settings) portion of the **Security** page. #### Example: Setting a Reverse Proxy Using an nginx Proxy It is the proxy's responsibility to ensure clients cannot spoof the header being used for the forwarded IP address. This example illustrates a proxy setting which includes reverse proxy settings. ``` location /arc/ { proxy_pass http://localhost:8080/; proxy_set_header Host $host; proxy_set_header Referer $http_referer; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Port $server_port; proxy_set_header X-NginX-Proxy true; proxy_redirect http://localhost:8008/ http://$host:8008/arc/; } # Declare all trusted IP or CIDR set_real_ip_from 192.168.1.100; set_real_ip_from 10.0.0.0/8; # Define the request header field whose value will be used to replace the client address real_ip_header X-Forwarded-For; # Enable recursive resolution to skip all trusted proxy IPs from X-Forwarded-For real_ip_recursive on # Optionally, set the real client IP to a new header Jetty_Forwarded-For: $remote_addr ``` This configuration prevents header spoofing by explicitly trusting only the specified proxy IPs (192.168.1.100 and 10.0.0.0/8). If a malicious user fabricates an `X-Forwarded-For` header, nginx ignores it and uses the IP closest to the client that came through the trusted proxy. # Dashboard Page Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/dashboard How to configure and use the CData Arc Dashboard to view metrics and data trends. The Dashboard is your CData Arc landing page. By default, the data refreshes every minute, but you can use the **Auto Refresh** list to set the refresh interval. The options are: * Every 10 seconds * Every 30 seconds * Every 1 minute * Every 5 minutes * Off Also by default, it displays metrics for the last 7 days, but you can use the calendar dropdown to select your preferred time period. The options are: * Today * Past 24 hours * Past 3 days * Past 7 days * Past 30 days * Past 3 months * Past 12 months > [!NOTE] > > The time periods shown vary based on your [cleanup interval settings](./settings/cleanup-settings), so you might not see all of the time periods listed above. The rest of the dashboard is divided into a number of sections, each of which is covered below. The dashboard is user-specific, so you can customize it to fit your needs by using the **Edit** button at the top right of the page, or the **Edit** links on certain cards. ## Video Resources Watch this short video to learn more about the Dashboard options. [https://www.youtube.com/embed/sysoNl6d-mY](https://www.youtube.com/embed/sysoNl6d-mY) ## Key Metrics ![Key metrics section of the dashboard](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dashboard_key_metrics_264.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=c2088b86fe762a3daf7fd1383d535117) You can display between two and eight metrics that provide a view into your application's health, volume, and usage. To pick your metrics, click the **Edit** button, then click the **Edit** link in the Key Metrics section. Choose from the following list: * Total Messages * Total Transactions * Total Transactions Sent * Total Transactions Received * Total MFT Transactions * Total EDI Messages * Total Errors * Total Send Errors * Total Receive Errors * Total Connectors * Active Connectors * Connectors in Backoff * Total Workspaces * Active Workspaces * Total Bytes Processed * Average Bytes Processed * Average Processing Time * Success Rate * Failure Rate * SLA Violations * At Risk SLAs ## Trends ![Trends section of the dashboard](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dashboard_trends_264.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=a698a37e20bb7e335a763519154f39dd) The Trends section helps to identify trends for message activity globally, as well as by Workspace or Connector Id. ### Messages This chart is a snapshot of the number of messages received over your chosen time period. It shows you how many were successful, and how many caused an error. Hover over the bars in the chart to see counts of the two statuses. ### Transactions by Workspace This table gives you a breakdown of the number of messages in each workspace, and it compares them to the previous time period. This lets you observe trends in the volume of messages that each workspace is processing. You can **Edit** this widget to select which workspaces to display. By default it includes all workspaces. ### Transactions by Connector Id This table gives you a breakdown of the number of messages each connector has processed. It also compares them to the previous time period so you can observe trends in the volume of messages each connector is processing. You can **Edit** this widget to select which connectors to display. By default it includes all connectors. ## EDI Documents ![EDI documents section of the dashboard](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dashboard_edi_documents_264.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=4fa34fd77ccb07fb3d894ef0e646b8d3) The EDI Documents section only appears if you have opted-in to the tracking of EDI transaction types. The **Track Transaction Types** checkbox is on the **Advanced** tab of some EDI connectors. When enabled, the application adds the transaction type as a tracked header on the message. The dashboard provides insight into the trends for between two and eight document types. You can **Edit** this widget to select which EDI Document types to display. Use the **Hide** toggle to remove this section from the dashboard. ## Recent Messages ![Recent messages section of the dashboard](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dashboard_messages_264.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=519b5d5adf5eab717064fc34f619c7cc) This is a simplified view of what you see on the **Activity > Messages** tab. It displays the 25 most recent messages. Click the **See all** link to go to the Messages tab and view the full list. See [Messages Tab](./activity#messages-tab) for details. # DMZ Gateway Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/dmz-gateway How to configure CData Arc to receive secure connections through a DMZ gateway using an SSH reverse tunnel. CData Arc can receive secure connections through a demilitarized zone (DMZ) gateway. Using a DMZ protects corporate firewalls and maintains network security by funneling all external connection attempts to the DMZ. > [!NOTE] > > This feature is currently only available if you use the embedded web server included with the .NET edition of Arc. ## How the Gateway Works Arc supports establishing an SSH reverse tunnel to receive data sent to the DMZ. Here's how it works: * An SSH server sits in the DMZ and transfers data between Arc and external trading partners. * Arc connects to this SSH server and opens an SSH reverse tunnel on any open port (for example, port 7777). * Once the tunnel has been opened, the SSH server forwards any traffic it receives on port 7777 directly to Arc. * Trading partners connect to the SSH server and send any data intended for Arc to port 7777. * The data is forwarded from the SSH server to Arc using the SSH standard for transport security. This configuration allows trading partners to send business data to Arc while only having access to the SSH server in the DMZ. ## Setting Up the Gateway Follow these steps to enable DMZ gateway support: 1. Install an SSH server in the DMZ or choose a DMZ with an SSH server already installed—for example, an Amazon Machine Image (AMI) pre-loaded with an SSH server, or a free OpenSSH server implementation installed on a DMZ machine. 2. Enable port forwarding in the SSH server. To do this, open the server's `sshd_config` file and set `GatewayPorts` to **yes**. 3. Right-click the Arc icon in the system tray, select **Server Options**, and navigate to the **Cloud Gateway** tab. ![Cloud Gateway configuration tab](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/CloudGateway.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=75c59d63c3e05bbacec4aebbb83010f4) 4. Check **Enable Cloud Gateway**. 5. In the **Server** section, configure the connection settings for the SSH server in the DMZ. 6. In the **Forwarded Port** section, set the **Forwarding Port** to the port on the SSH server to use when forwarding data to Arc. You can also forward two additional ports to establish tunnels for SFTP and OFTP ports. See [Additional Forwarding Ports](#additional-forwarding-ports) for details. 7. Click **Test Connection** button to verify that the connection is successful. 8. Restart the Arc embedded web server to automatically open an SSH reverse tunnel on the specified port. After the above steps are complete, provide trading partners with connection details to the SSH server in the DMZ. Instruct them to send data to this SSH server on the port specified by **Forwarding Port** instead of the default port (22). ### Additional Forwarding Ports Configuring the **Forwarding Port** allows traffic to be forwarded for the HTTP traffic used in Arc. You can also forward two other ports to establish tunnels for SFTP and OFTP ports. > [!NOTE] > > The DMZ Gateway supports a total of three forwarded ports for traffic, one of which is the HTTP or HTTPS port used to access HTTP or HTTPS-based service endpoints. While you can use the OFTP and SFTP ports for any other listening service on the machine, support for FTP Server connections requires an additional port range beyond the command channel. This means that FTP Server connections are not supported. To forward additional ports, add the following four lines to the `CData.Arc.xml` configuration file in the application data directory (default path: `C:\ProgramData\CData\Arc\CData.Arc.xml`): ```xml theme={null} 6619 6619 22 2022 ``` `Port` is the port that the traffic is forwarded to and `RemotePort` is the port to open on the SSH server. > [!NOTE] > > You cannot forward the port that is already being used for the SSH service. For example, if the SSH server is listening on port 22, the `SFTPRemotePort` cannot also be port 22. ## Maintaining the Gateway Arc automatically opens the gateway when the server (re)starts. Arc also handles reconnecting to the SSH server if the connection is dropped for any reason. No user maintenance is required to keep the gateway up and running. # Git Version Control Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/git-version-control How to use Git-based version control for CData Arc configuration management, including installation, setup, and workflow examples. CData Arc supports Git-based version control for most of your application configurations, bringing enterprise-grade change management to your data integration platform. This feature enables teams to track configuration changes with full audit trails, maintain separate environments for testing and production, and quickly recover from issues by rolling back to known good states. With Git integration, team members can collaborate on Arc configurations with proper attribution and change history, ensuring compliance requirements are met while reducing the risk of configuration errors. Organizations can apply the same rigorous version control practices to their Arc configurations that they already use for application code. Arc supports Git version control with a **Remote** repository connected to a hosted Git platform (such as GitHub, GitLab, Azure DevOps, or Bitbucket). Self-hosted installations also support a **Local** repository that runs directly on the Arc server, without requiring any external Git infrastructure. By default, a single **Global** repository covers the entire Arc instance. You can also give an individual workspace its own separate repository. See [Global and Workspace Repositories](#global-and-workspace-repositories) for details. > [!NOTE] > > Arc tracks your configuration files (such as `*.cfg`, XML, JSON, and binaries), with two exceptions: the [vault](../administration/settings/global-settings-vault) and its contents, and [users and roles](../administration/settings/user-roles). To avoid versioning sensitive values like tokens, usernames, and passwords, CData recommends that you store *all* sensitive values in the vault and reference the vault items in connectors where the sensitive values are required. See [Architecture](../architecture) for a complete list of tracked file types. When a workspace has its own repository, that workspace's files are tracked in the workspace repository instead of the global repository. > > To exclude files from version control, add entries to a `.gitignore` file in the root of your Arc [Application Directory](../glossary). To exclude a file or pattern, add it on a new line in `.gitignore`. Common candidates for exclusion include environment-specific configuration files that differ across deployments and large binary files that would inflate repository size. Keep in mind that excluded files are never committed, pushed, pulled, or restored, so you must manage them separately in each environment. ## Installation and Configuration ### Prerequisites Before you can use the version control features, you must install Git on all Arc environments, including development and production. If you are using a remote repository, you must also create it before configuring version control. Arc supports all major Git platforms, including GitHub, GitLab, Azure DevOps, and Bitbucket. If you are using a local repository, no remote setup is required. #### Installing Git ##### Windows 1. Download the Git installer from [git-scm.com/download/win](https://git-scm.com/download/win). 2. Run the installer and follow the installation wizard using your preferred settings. 3. Verify the installation by opening a command prompt or PowerShell and running: ``` git --version ``` You should see output showing the installed Git version. ##### Linux Debian/Ubuntu Run the following: ``` sudo apt-get update sudo apt-get install git ``` ##### Linux Fedora/RHEL/CentOS Run the following: ``` sudo dnf install git ``` Run this command to verify the installation: ``` git --version ``` Once Git is installed and verified, you can begin to use the Arc version control features. ### Configure Version Control Version control can be configured at two levels: for the application as a whole (the global repository) and, optionally, for individual workspaces. See [Global and Workspace Repositories](#global-and-workspace-repositories) to understand how the two relate. #### Configure the Global Repository Navigate to **Settings > Advanced > Git Version Control** to set up version control. See [Git Version Control Settings](../administration/settings/git-version-control-settings) for details. When you click **Configure**, you first choose a **Configuration Type**: * **Remote**: connects to a hosted Git repository. This enables push, pull, and team collaboration across multiple Arc environments. * **Local**: creates a standalone Git repository directly on the Arc server. No remote URL or credentials are required. This is the fastest way to get started with version control. For remote repositories, supply the repository URL and authentication credentials. Creating the remote repository itself is beyond the scope of this documentation; consult your Git platform's documentation. For example, when using HTTPS authentication with GitHub, you must use a Personal Access Token (PAT) instead of a password. #### Configure a Workspace Repository To version an individual [workspace](../../flows/workspaces) in its own repository: 1. Open the workspace, then click **Settings** in the ellipsis menu. Click the **Git Version Control** tab. 2. Toggle on the **Enable Git Version Control** switch. If the global repository is set up, a warning explains that enabling version control for the workspace removes it from the global repository, so it is tracked only in the repository you configure. ![Workspace settings Git Version Control tab with version control enabled](/26.4/images/version_control_workspace_settings_enabled.png) 3. Configure the repository. The fields are the same as those for the application-level repository; see [Git Version Control Settings](../administration/settings/git-version-control-settings#workspace-git-version-control-settings) for details. Select a **Configuration Type**: * **Remote**: connects to a hosted Git repository. Provide the repository URL and authentication credentials. * **Local**: creates a standalone Git repository on the Arc server. No remote URL or credentials are required. For a remote repository, provide the repository URL and authentication credentials. 4. Click **Save**. > [!NOTE] > > Workspace repositories follow a few rules, such as a restriction on the **Default** workspace. See [Global and Workspace Repositories](#global-and-workspace-repositories) for details. > > You can convert a workspace repository between Local and Remote (see [Migrate from a Local to a Remote Repository](#migrate-from-a-local-to-a-remote-repository) and [Convert a Remote Repository to a Local Repository](#convert-a-remote-repository-to-a-local-repository)). > > You can change the remote URL after you save the configuration (see [Edit the Remote URL](#edit-the-remote-url)). ## Global and Workspace Repositories Arc supports two kinds of repository: * **Global**: the application-level repository that you configure under **Settings > Advanced > Git Version Control**. By default, it tracks the configuration for the entire instance, including every workspace. * **Workspace**: an optional repository that versions a single workspace, separate from the global repository. A repository that has no remote configured is labeled **Local** in the Arc interface, regardless of whether it is the global repository or a workspace repository. Keep the following in mind when you use workspace repositories: * If the global repository is set up, enabling a workspace repository moves the workspace out of the global repository. After you enable it, the workspace is tracked only in its configured repository. * Shared resources stay in the global repository. Connections, certificates, and profiles that a workspace uses remain in the global repository. A workspace repository is therefore not a self-contained backup of everything that the workspace needs. * Workspace repositories are independent of the global repository. You can enable a workspace repository whether or not you have set up a global repository. * The **Default** workspace cannot have its own repository. Its **Git Version Control** tab shows the message *A separate Git repository is not available for the Default workspace.* ### One Repository at a Time Every version control action applies to exactly one repository: the repository you currently have selected. Commit, push, pull, branch, discard, restore, and status all apply to the selected repository and nothing else. No action spans multiple repositories. * An action that you start from a row in the repository list stays with that row's repository, even if the active workspace changes while the action runs. * Progress, errors, and counts are tracked separately for each repository, so a failure in one repository does not block work in another. Which repository is selected by default depends on where you are in Arc. On the **Flows** page, the selection follows the workspace that you are working in. Everywhere else, the selection is the global repository. You can always change the selection (see [Select a Repository](#select-a-repository)). ## Development Environments > [!NOTE] > > This section applies to **remote** repositories only. Local repositories do not have a remote to sync with, so separate development environments are not required. When you use version control with Arc in a shared production environment, CData recommends setting up individual development environments for each team member. This provides users with the best experience and avoids conflicts that can occur when multiple users try to push and pull changes simultaneously in a shared environment. Working directly in a shared production environment with version control can lead to: * Merge conflicts when multiple users make changes at the same time * Poor user experience due to competing version control operations By using development environments, each team member can work independently, test their changes locally, and push completed work to the repository without interfering with others or disrupting production operations. Before you begin, make sure that master key encryption is enabled on both the production and development instances. If it is not, the data encryption key export and import in this procedure fail with the error *Master key encryption is not enabled*. This applies to existing installations as well as new ones. See [Check Whether Master Key Encryption Is Enabled](../installation/custom-master-key#check-whether-master-key-encryption-is-enabled) to check each instance and [Enable Master Key Encryption](../installation/custom-master-key#enable-master-key-encryption) to enable it. Follow these steps to configure a local development environment that mirrors your production Arc instance: 1. Export the data encryption key from production (see [Understanding Data Encryption Keys and Master Keys](../installation/custom-master-key#understanding-data-encryption-keys-and-master-keys) for more information on key usage) 1. In your production environment, use your API key to call the [Admin API exportDataEncryptionKey](../../admin-api/admin-api-actions) endpoint. Use the `passphrase` parameter to create the passphrase that will be used when you import the key to the development environment. 2. Save the response because you need this data encryption key for your development environment. 2. Export the vault from your production environment: 1. Use the UI 1. Navigate to **Settings > Advanced > Migration > Export** 2. Select **Custom Export** and create a **Password**, then click **Next** 3. On the **Settings** tab, only check **Vault** 4. Click **Export** to perform the export and download the file 2. Use the [Admin API export](../../admin-api/admin-api-actions) endpoint with the vault option enabled 3. Install Arc in your development environment (if it is not already installed) 1. Install Arc on your development machine 2. Complete the initial setup and create your first [user account](../administration/settings/users) 3. Install your development license 4. Create an API key for API access 4. Import the data encryption key to development: In your development environment, use your API key to call the [Admin API importDataEncryptionKey](../../admin-api/admin-api-actions) endpoint with the *same passphrase* you used when you exported the key from production. This ensures that your development environment uses the same data encryption key and can decrypt the vault contents. > [!WARNING] > > Importing the data encryption key replaces the development instance's data encryption key. Values that were encrypted on that instance before the import can no longer be decrypted. Import the key before you enter any connection credentials or other sensitive connector settings. 5. Import the vault to development 1. Use the UI 1. Navigate to **Settings > Advanced > Migration > Import** 2. Upload the vault file you exported earlier 3. Complete the import process 2. Use the [Admin API import](../../admin-api/admin-api-actions) endpoint to upload and import the vault file. 6. [Pull](#pull-changes) the remote repository to sync your development environment with the production configuration. If any workspaces in production have their own repositories, configure and pull each of those repositories as well. Shared resources such as connections, certificates, and profiles remain in the global repository, so you must pull the global repository too. 1. Use the UI 1. Install Git by following the instructions in [Prerequisites](#prerequisites) 2. Navigate to **Settings > Advanced > Git Version Control** 3. Configure your remote repository connection (see [Git Version Control Settings](../administration/settings/git-version-control-settings) for instructions) 4. Pull the latest changes from the remote repository 2. Use standard Git commands to configure the repository connection on your local machine Once your development environment is configured, you can: * Make and test changes locally without affecting production * Commit changes to your local repository * Push completed work to the remote repository * Pull updates made by other team members * Follow standard Git workflows (such as branches and pull requests) for collaborative development Your production environment can now pull approved changes from the remote repository on a controlled schedule. ## Version Control Operations Arc's Git integration provides essential version control operations that allow you to manage configuration changes, synchronize with remote repositories, and maintain multiple development branches. These operations follow standard Git workflows while being tailored specifically for Arc's configuration management needs. Once version control is implemented on your environments, the standard Arc footer becomes a status bar that displays Git metrics and actions such as the number of local files changed, the number of remote commits, the selected repository, and the current branch. All of these metrics and actions apply to the selected repository (see [One Repository at a Time](#one-repository-at-a-time)). ![Git version control footer toolbar showing the selected repository](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_footer_toolbar.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d59c1b1cf3d5ea75acdc08db5941353f) ### Permissions Version control is restricted to administrators. Only users with the [Admin role](../administration/settings/user-roles#admin-role) can use version control features, including viewing version control logs, for both the global repository and workspace repositories. The Version Control options are hidden in the UI for all other users. ### Commit and Push *Commit* captures all configuration changes in the selected repository as a single atomic unit and saves them to that repository. When you commit, Arc automatically records your Arc username as the author and timestamps the change, ensuring full traceability of who made what changes, and when. A commit message is required to document the purpose of your changes. *Push* synchronizes your local commits to the remote Git repository, making your changes available to other team members and backup systems. If the remote branch contains newer changes that you haven't pulled yet, the push operation fails with an error message and suggestions of what to do next. This prevents overwriting other users' work and ensures proper merge handling. For remote repositories, commit and push are handled in a single action. See [Commit and Push New Changes](#commit-and-push-new-changes) for details. For local repositories, there is no push step; the toolbar shows **Commit** instead of **Commit and Push**. ### Pull > [!NOTE] > > Pull is available for remote repositories only. Local repositories do not have a remote to pull from. *Pull* retrieves the latest changes from the selected repository's remote and updates your local Arc configuration. Arc does not pull automatically in the background. Before pulling, you must commit or discard any uncommitted local changes; the operation fails if uncommitted changes are detected. If there are merge conflicts between your local version and the remote version, the pull fails with a message directing you to resolve the conflict externally using Git tools (see [Manage Commit and Pull Conflicts](#manage-commit-and-pull-conflicts)) for details. Pull operations can be initiated from the Git toolbar. See [Pull Changes](#pull-changes) for details. ### Select a Repository When more than one repository is initialized, the toolbar includes a searchable repository list. Use it to see the state of every repository and to choose which one is selected. ![Repository list showing the global repository and a workspace repository](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_repository_list.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=4caa8b6cd1106ec27e4a1fbfde7ef242) For each repository, the list shows: * The Git provider and the repository name * The scope: **Global**, or the name of the workspace * The current branch * The number of local changes * The number of incoming commits A repository with no remote is labeled **Local** in the scope. Each row in the list provides the following actions: * **Commit & Push** (or **Commit** for a repository with no remote): opens the [Local Changes](#commit-and-push-new-changes) flyout with that repository preselected. * **Pull**: pulls the incoming commits for that repository. * **Select**: makes that repository the selected repository. The current repository shows **Selected** instead. Actions that you start from a row always apply to that row's repository. If a repository is in conflict, its row shows the message *Conflicts within repo. Please go to repo to resolve conflicts.* Open that repository to resolve the conflict (see [Manage Commit and Pull Conflicts](#manage-commit-and-pull-conflicts)). ### Branch Branch management allows you to maintain separate configuration versions for different purposes, such as development, testing, and production environments. Branches belong to a single repository, so the branches that you see and the branch changes that you make apply to the selected repository only. Use the branch dropdown in the toolbar to switch between branches, create new ones, or delete stale ones. When creating a new branch, specify the branch name and select which branch to create it from (the default is to use your current branch). This enables you to experiment with configuration changes without affecting your main environment. See [Create Branches](#create-branches) and [Delete Branches](#delete-branches) for details. ### Logging All version control changes are captured in the [Audit logs](../administration/activity#audit-logs) section of the [Activity](../administration/activity) page. You can download logs from that page as necessary. ![Audit logs for version control changes](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/version_control_audit_logs.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=7337feb661705e740369380d4e9b97c8) ## History Viewer Once version control is enabled, a **Commits History** tab appears on the **Activity** page. It shows the commit history of one repository at a time and provides details such as the commit hash, commit message, author, date and time, and a list of the files that were changed in each commit. You can search and filter the content on this tab. ![Commit history tab on the Activity page](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_commit_history.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=bd911cdc542733a0a575f43d3c6098b0) * Use the **Repository** filter to choose which repository's history the table shows. The tab shows the history of a single repository, not a combined history of all repositories: the table does not include a repository column, and there is no option to show all repositories at once. Actions on a row, such as pull and restore, apply to the repository whose history you are viewing. If no repository is configured, the tab shows *No repositories available*. * Use the **All-time** and **All Users** filters to further narrow your results. Click a **Commit** hash to open the **Commit Details** page. It opens to a **File Changes** tab which displays a diff view of the files in the commit so you can see exactly what has changed. Click the **Details** tab to see the author, commit message and timestamp. ![Commit details with file diff view](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_commit_details.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=b5b14f70fc7bd51fdc0ac939398b01c2) The diff displays the old and new values of all changed settings, with the following notes: * Passwords and secrets are masked * Vault references are prefixed with a key symbol Click **View with all settings** in a section to see the full list of settings in that section, including those that were not changed in the commit. ![Commit details showing all settings](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_commit_details_all_settings.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=10831e2d346ece0e1f850dc6a00df293) The **Go To Resource** link on the **File Changes** tab takes you to the part of Arc where the file was changed. For example, if you are viewing the diff of a [MySQL](../../connectors/mysql) connector, clicking the link takes you to that MySQL connector. ## Version Control Workflows Examples The following examples demonstrate common version control scenarios encountered when working with Arc's Git integration. These workflows cover the essential operations: pulling updates from remote repositories, creating and deleting branches, committing and pushing changes, restoring changes, resolving conflicts when local and remote changes diverge, and discarding unwanted modifications. ### Pull Changes > [!NOTE] > > This workflow applies to remote repositories only. Pull applies to the selected repository. To pull a different repository, either select it first or click **Pull** on its row in the [repository list](#select-a-repository). You can pull changes from remote in several ways: * Click **Pull** on the toolbar to pull the commits without reviewing them * Click the number of commits on the toolbar to open the **Pull Remote Commits** preview ![Pull remote commits preview](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_pull.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=16bc355aee1740236ce5a36d32d64fcd) * Navigate to the **Commits History** tab on the **Activity** page and click the link to the number of remote commits on the left or the **Pull** button on the right to open the **Pull Remote Commits** preview The **Pull Remote Commits** preview lists all commits that are ready to be pulled. Click a commit to see its details, then hover over the number of files to see which files are included in the commit. To complete the task, click **Pull Commits**. ![Pull remote commits confirmation dialog](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_pull_remote_commits.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d3490b81925f1f5ff18bee2a225762d2) ### Create Branches To create a new branch, click the branch icon on the toolbar (you can also switch between branches here). ![Branch switcher on the toolbar](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_switch_branch.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=5bba02d3e01e1ef189ff000e622160a2) Click **Add Branch** and give your branch a **Name**. Choose which branch to copy from, then click **Add**. ![Add branch dialog](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/version_control_add_branch.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=4ecaf731a9fab5fc69a96654e3838f56) ### Delete Branches To delete a branch, click the branch icon on the toolbar. Hover over the branch you want to delete and click the trashcan icon. ![Delete branch dialog](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_delete_branch.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=fd1cbe700050379ada97f0b82c21d074) ### Commit and Push New Changes When you are making changes in Arc, the toolbar shows the total number of files that have been changed in the selected repository: ![Git toolbar showing number of changed files](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_footer_toolbar.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d59c1b1cf3d5ea75acdc08db5941353f) Click **Commit & Push** to open the **Local Changes** flyout. It shows details about the files that you have changed in the selected repository. You must supply a commit message before you can commit and push your changes to remote. The flyout header contains a repository selector that shows the name, scope (**Global** or workspace name), and branch of the repository that you are committing to. It also provides refresh and help icons, and a button that shows how many of the changed resources are selected for the commit. Use the checkboxes in the **Select Resources** list to choose which changed resources to include. ![Local Changes flyout with the repository selector in the header](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_local_changes_flyout.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=a03cb0a7204e6c70c589241ba46ea3e1) To commit to a different repository, click the repository selector in the header and click **Select** next to the repository that you want. The changed files, branch, status, and commit message update to match the repository that you chose. ![Repository selector open in the Local Changes flyout](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_local_changes_repository_list.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=f014d66af9546299fd6495122efcfa25) > [!NOTE] > > For remote repositories, commit and push happen as a single step; you do not commit locally and then push separately. For local repositories, only **Commit** is available since there is no remote to push to. After you commit and push the changes, a confirmation message in the toolbar lets you know the result of your action. ### Restore Changes from a Previous Revision To restore resources to their previous state, navigate to the **Commits History** tab on the **Activity** page and use the **Repository** filter to choose the repository that contains the resources. Restore applies only to that repository. Click the ellipsis to the right of the **Files Changed** column, then click **Restore**. Review the diff between the current state and the proposed restore point state, then click **Next**. Be aware that if you proceed with the restore, any uncommitted changes in those files are permanently lost. Click **Restore Resources** to complete the task. ### Manage Commit and Pull Conflicts > [!NOTE] > > This section applies to remote repositories only. Commit conflicts do not occur with local repositories because there is no remote branch to diverge from. Conflicts can arise when you have local changes to push and there are also remote changes to pull. Conflicts are tracked per repository: a conflict in one repository does not block actions in another. The toolbar reflects the conflicts, and provides suggestions on how to proceed. The same error messages also appear on the **Commit** and **Pull Remote Commits** preview pages. In the [repository list](#select-a-repository), a repository in conflict shows the message *Conflicts within repo. Please go to repo to resolve conflicts*. ![Commit and pull conflict message in toolbar](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_commit_pull_conflict.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=547294a385ffb1bf84653946ac11f754) One option is to manage the conflicts manually, and the other is to commit the current changes to a different branch, which then allows you to pull remote changes to the current branch. Click **Please commit to a new branch** to open a page where you can select a different branch to commit to. Once you select (or create) the new branch, you can push your changes using **Commit & Push Changes**. ### Discard Changes To discard your current changes in the selected repository, click the trashcan icon in the toolbar. This opens a **Discard Changes** pane, where you can select exactly which changes to discard. Click **Next**, confirm that your selections are correct, then click **Discard Changes**. Once complete, a confirmation message appears and the number of changes in the toolbar is reduced by the number of changes that were discarded. You can also discard changes on the **Local Changes** flyout. Select the resources to discard, then use the dropdown arrow on the **Commit...** button and choose **Discard Changes**. Review your selections, then click **Discard Changes**. ## Migrate from a Local to a Remote Repository If you have a local repository and want to connect to a remote server while preserving your commit history, you can convert it to a remote repository. This works for global and workspace repositories. 1. Create the remote repository: set up a new, empty repository on your Git platform (such as GitHub, GitLab, or Bitbucket). 2. Open the version control settings: for the global repository, navigate to **Settings > Advanced > Git Version Control**. For a workspace repository, open the workspace, then click **Settings** in the ellipsis menu. Click the **Git Version Control** tab. 3. Change the Configuration Type from **Local** to **Remote**. 4. Enter the remote repository URL and your authentication credentials. See [Git Version Control Settings](../administration/settings/git-version-control-settings) for the available authentication schemes. 5. Click **Save**. Arc adds the remote to the repository and applies your credentials. Once your **Status** shows **Connected**, click **Commit & Push** to send your existing local commits to the remote repository. Your local commit history and tracked files are preserved. Arc does not recreate the underlying `.git` directory. ## Convert a Remote Repository to a Local Repository To stop syncing a repository with its remote while keeping its history, open the version control settings for the global or workspace repository, change **Configuration Type** from **Remote** to **Local**, and click **Save**. Arc cleanly removes the remote from the repository. Your local commit history and tracked files are preserved, and **Pull** is no longer available for that repository. ## Edit the Remote URL You can change the repository URL after version control is initialized. To change the remote URL for a repository that is already connected to a remote: 1. Open the version control settings for the repository. For the global repository, navigate to **Settings > Advanced > Git Version Control**. For a workspace repository, open the workspace, then click **Settings** in the ellipsis menu. Click the **Git Version Control** tab. 2. Edit the repository URL and, if required, your credentials. 3. Click **Save**. Arc updates the remote and applies the credentials. Your local commit history and tracked files are preserved. # Importing and Exporting Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/importing-exporting-flows How to import and export connectors, flows, workspaces, and global application data in CData Arc. You can import and export individual [connectors](../../connectors), [flows](../../flows), [EDI TPMC](../../tpmc) partners and documents, [workspaces](../../flows/workspaces), and global application data in CData Arc. You import and export these components using the following file types: * `.arcflow` files: Store exported flows, connectors, or the contents of a workspace that you can import into different instances of Arc. When you export a whole workspace, the file also includes workspace-level settings such as connections, certificates, profiles, and vaults. This file type stores only one workspace at a time. Use **Export Flows** and **Import Flows** to create and import `.arcflow` files. * `.arc` files: Store global application data that you can import into different instances of Arc. This file can contain multiple workspaces and settings as well as users, TPMC partners and documents, reports, or other global data. To create `.arc` files, use **Export Workspace** on the workspaces grid. To import existing `.arc` files, use **Import Workspace** in the **Add Workspace** dropdown. You can also use the **Export** and **Import** buttons in **Settings > Advanced > Migration**. The export and import controls pair up as follows: | File type | Export | Import | | - | - | - | | `.arcflow` | **Export Flows** | **Import Flows** | | `.arc` | **Export Workspace** (workspaces grid) | **Import Workspace** (**Add Workspace** dropdown) | | `.arc` | **Export** (**Settings > Advanced > Migration**) | **Import** (**Settings > Advanced > Migration**) | ## Importing This section provides steps for importing your data using various methods: using drag-and-drop functionality, using a `.arcflow` file, and using a `.arc` file. It also explains how to resolve conflicts in your files, import sensitive data, and specify the data that you want to import. ### Importing with Drag-and-Drop Functionality If you drag a `.arcflow` file into a flow canvas, the application imports the connectors in the `.arcflow` file. After you import the file and refresh the page, the new connectors appear at positions in the flow according to the imported settings. If the new connectors are not immediately visible, use the **Search** bar in the workspace to find them. ### Importing Connectors, Flows, or a Workspace To import a connector, flow, or workspace: 1. Click the **More Options** ellipsis in an existing workspace, then select **Import Flows** to open the **Workspace Import** dialog. ![Workspace menu with Import Flows selected](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/import_workspace_settings_icon.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=69e9d14b8d633c6cff536d1fa2376d18) 2. Select the file to import. Then, configure the import options that you want. See [Setting Import Options](#setting-import-options) for more information on these settings. ![Workspace import dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/workspace_import_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=7ce2338e5dd58dfb80e5eead148c3c9a) 3. Click **Next** to start the import process. After you import the file and refresh the page, the new connectors appear at positions in the flow according to the imported settings. If the new connectors are not immediately visible, use the **Search** bar in the workspace to find them. ### Importing Multiple Workspaces and Global Application Data You can import a `.arc` file from the workspaces grid or from the **Settings** page. #### Importing from the Workspaces Grid To import a `.arc` file from the workspaces grid: 1. On the **Flows** page, use the **Add Workspace** dropdown to select **Import Workspace** and open the **Import** dialog. 2. Navigate to the `.arc` file you want to import and choose **Import**. Then, configure the import options that you want. See [Setting Import Options](#setting-import-options) for more information on these settings. 3. Click **Import** to check for conflicts and to complete the import. See [Resolving Import Conflicts](#resolving-import-conflicts) for more information. #### Importing from the Settings Page To import multiple workspaces and global application data from the **Settings** page: 1. Click the [Settings](./settings/settings-page) gear icon in the navbar. 2. Click the **Advanced** tab, scroll to the [Migration](./settings/migration) section, and choose **Import**. This enables you to perform a full import of the application, which includes all of the application's data and settings. ![Settings migration section](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_migration.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=01d29e290187d7ea54398a3df2cad50b) 3. Select the file to import. Then, configure the settings that you want. See [Setting Import Options](#setting-import-options) for more information on these settings. ![Import dialog for a .arc file](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/application_settings_import_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=22a73a313fb0563805a7823dc8698bfb) 4. Click **Import** to start the import process. > [!NOTE] > > If there are any conflicts in your file, the **Resolve Import Conflicts** dialog opens. See [Resolving Import Conflicts](#resolving-import-conflicts) for more information. ### Setting Import Options When you import data, set the following options: the file to import, what data to import, and whether to import sensitive data. * **File:** Select the `.arc` or `.arcflow` file to import. * **Type:** Select the type of import. * **Import all workspaces and settings:** Imports everything that your file contains. This is the default **Type**. * **Custom import:** Enables you to select which components to import in the **Customize Workspace Import** dialog (for `.arcflow` files) or the **Custom Import** dialog (for `.arc` files). For example, the following image shows the **Customize Workspace Import** dialog. The **Custom Import** dialog is similar. ![Customize workspace import dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/customize_workspace_import_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=0a986cee60dc4874a6d991d9be3335cf) * **Import Sensitive Values:** To import sensitive values, check **Enable** in the **Import Sensitive Values** section of the **Workspace Import** dialog (`.arcflow` files) or in the **Import** dialog (`.arc` files). ![Import dialog for a .arc file with sensitive values](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/application_settings_import_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=22a73a313fb0563805a7823dc8698bfb) When this is enabled, you must provide a password. When you enter a password, Arc decrypts the sensitive values that are saved in port.cfg files, adds them to the export file, and encrypts them with the provided password. * If you provide a password that does not match that of the file, Arc displays an error and does not import any information. * If you do not enter a password, Arc masks and excludes sensitive values from the export file. ### Resolving Import Conflicts When you import a file, it might contain conflicts. If your file contains conflicts, the **Resolve Import Conflicts** dialog is displayed. ![Resolve import conflicts dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/resolve_import_conflicts_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=b73d18c1f6edc4a1444039d0be0ba28e) You have three ways to resolve the conflicts: * **Don't import any conflicts:** Resources in the imported file do not overwrite anything that already exists in the application. This is the default. > [!NOTE] > > This might cause imported flows not to function as expected. * **Replace all conflicts:** All the resources in the imported file are accepted, overwriting any resources that might already exist. > [!NOTE] > > This might cause overwritten flows not to function as expected. * **Manually resolve conflicts:** This opens another dialog where you select exactly which resources to replace and which resources to skip. The options depend on the file type: * For `.arcflow` files, select **Replace** or **Don't Import** for each resource. * For `.arc` files, select **Replace Existing** for each resource you want to replace. If you select either of the first two methods, click **Import**. If you select the third method, the dialog that opens when you click **Import** depends on the file type: * **.arcflow files:** The **Review Flow Import Conflicts** dialog is displayed. ![Review flow import conflicts dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/review_flow_import_conflicts_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=63804eae9b5e5dd778cd785ccca4c851) * **.arc files:** The **Import Conflict Resolution** dialog is displayed, with the conflicts listed by type. Work through each section and manually decide on each object. ![Import conflicts listed by type for a .arc file](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/manually_resolve_import_conflicts.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=fad0e8394faf1a19a81e06cc54a52630) After you manually resolve the conflicts, click **Import**. ## Exporting This section provides steps for how export individual connectors, flows, TPMC partners and documents, and workspaces. It also explains how to export sensitive data and specify the data that you want to export. ### Exporting an Individual Connector To export individual connectors in a `.arcflow` file: 1. Right-click a connector in a flow and select **Export Connector Settings**. 2. Configure the export options in the **Selected Items Export** dialog. See [Setting Export Options](#setting-export-options) for more information. ![Connector selected items export dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/connector_selected_items_export.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=82dbeb999c6d4039054546c9c29b2720) 3. Click **Export**. The resulting `.arcflow` file contains the exported connector. ### Exporting a Flow To export multiple connectors in a workspace: 1. Enable multi-select mode via the **Multi-Select** toggle. ![Multi-select button](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/select_multiple_item_button.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=7bc1c89d69a422b10073bb23bcd0a1d0) 2. Click and drag the multi-select cursor in the flow to create a box that contains each of the connectors that you want. 3. Release the cursor to select the connectors in the box. 4. Right-click a selected connector and select **Export Connector Settings**. 5. Configure the export options in the **Selected Items Export** dialog. See [Setting Export Options](#setting-export-options) for more information. ![Connector selected items export dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/connector_selected_items_export.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=82dbeb999c6d4039054546c9c29b2720) 6. Click **Export**. ### Exporting a Workspace as a `.arcflow` File To export a workspace as a `.arcflow` file: 1. Click the **More Options** ellipsis in an existing workspace, then select **Export Flows** to open the **Workspace Export** dialog. ![Workspace menu with Export Flows selected](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/export_workspace_settings_icon.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=4ffab8be9bd705ba59db171a44792193) 2. Configure the export options that you want. See [Setting Export Options](#setting-export-options) for more information. ![Workspace export dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/workspace_export_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=76caf4df64dfdda0b81a66996223ef51) 3. Click **Export**. The resulting `.arcflow` file contains the connector-settings files for the exported workspace, along with the workspace-level settings (such as connections, certificates, profiles, and vaults). ### Exporting Multiple Workspaces and Global Application Configuration Data You can create a `.arc` file from the workspaces grid, which exports only the workspaces you select, or from the **Settings** page, which exports the full application, including global data such as EDI TPMC partners and documents. #### Exporting Selected Workspaces To export only selected workspaces: 1. On the [Workspaces](../../flows/workspaces) grid, select one or more workspace cards. 2. Click **Export Workspace**. This button is available only when at least one workspace card is selected. 3. Configure the export options that you want. See [Setting Export Options](#setting-export-options) for more information. 4. Click **Export**. The resulting `.arc` file contains the selected workspaces. #### Exporting the Full Application To export all workspaces, EDI TPMC partners and documents, and global application data: 1. Click the [Settings](./settings/settings-page) icon on the navbar. 2. Click the **Advanced** tab, scroll to the [Migration](./settings/migration) section, then choose **Export**. This tab enables you to perform a full export of the application, which includes all of its data and settings. ![Settings migration section](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_migration.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=01d29e290187d7ea54398a3df2cad50b) 3. Configure your settings. See [Setting Export Options](#setting-export-options) for more information. > [!NOTE] > > For global exports, **Export Sensitive Values** is selected by default, and you must provide a password. ![Export dialog for a .arc file](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/applications_settings_export_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=1f0fabbadf3e50d9719fffad1e7fe135) 4. Click **Export**. The resulting `.arc` file contains the workspaces and global application data. ### Setting Export Options When you export data, set the following options: which file to export, what data to export, and whether to export sensitive data. * **File:** Select the `.arc` or `.arcflow` file that you want to export. * **Type:** Select the type of export. Choose whether to export all workspaces and data, or to create a custom export: * **Export all workspaces and settings:** Exports everything in your file. This is the default **Type**. * **Custom export:** Select the components to export in **Customize Workspace Export** (`.arcflow` files) or **Custom Export** (`.arc` files). For example, the following image shows the **Customize Workspace Export** dialog. The **Custom Export** dialog is similar. ![Customize workspace export dialog](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/customize_workspace_export_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=2e6fb9ff9820e9035d0b97f8cb0a6156) > [!NOTE] > > * [Users](./settings/users) are only exported in `.arc` files. They are excluded from `.arcflow` files. > * Views are only exported if all connectors in the workspace are included in the export. * **Export Sensitive Values:** To export sensitive values, check **Enable** in the **Export Sensitive Values** section of the **Workspace Export** dialog for `.arcflow` files. ![Workspace export dialog with sensitive values](https://mintcdn.com/cdata-arc/DSF8W5wN-d7O_6nX/26.4/images/workspace_export_dialog.png?fit=max\&auto=format\&n=DSF8W5wN-d7O_6nX\&q=85\&s=76caf4df64dfdda0b81a66996223ef51) When this is enabled, you must provide a password. > [!NOTE] > > When you export `.arc` files, you must provide a password. This option is enabled by default. When you enter a password, Arc decrypts the sensitive values that are saved in port.cfg files, adds them to the export file, and encrypts them with the provided password. **For `.arcflow` files only:** If you do not enter a password, Arc masks and excludes sensitive values from the export file. # Profiles Page Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/profiles How to configure global connector profiles for AS2, AS4, SFTP Server, OFTP, and other protocols in CData Arc. Certain connectors require configuring a profile before you can establish a connection. This profile contains partner connection settings that are used globally by CData Arc. These typically include identifiers that refer to your company as an entity, and private certificates. The settings on the **Profiles** page represent the values that are constant across all connections. The following connectors require a profile. Click the links to access configuration details for your connector: * HTTP * [AS2](../../connectors/as2#profile-configuration) * [AS4](../../connectors/as4#profile-configuration) * [GISB](../../connectors/gisb#profile-configuration) * [RosettaNet](../../connectors/rnif#profile-configuration) * Servers * [FTP Server](../../connectors/ftp-server#profile-configuration) * [HL7 MLLP](../../connectors/hl7mllp#profile-configuration) * [OFTP](../../connectors/oftp#profile-configuration) * [SFTP Server](../../connectors/sftp-server#profile-configuration) ## Profile Configuration The **Profiles** page is divided into two sections: **HTTP** and **Servers**. HTTP includes profiles for connectors that send and receive traffic over the HTTP protocol. Servers contains profiles for connectors that send and receive traffic over a different protocol (for example, SFTP Server uses the SSH protocol). ![Profiles page](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=01b6430d903a130428c56dc0a039676c) If you have not previously configured a profile, click the **Configure** button on a card to be guided through each step required to complete the configuration. For example, configuring the AS2 profile has three steps: 1. Provide the AS2 identifier. ![AS2 profile configuration step 1](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-config-as2-1.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=655c7c5e1bd02366309552e152668365) 2. Define the private certificate. ![AS2 profile configuration step 2](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-config-as2-2.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=bd10877141977464cf8b62179213afbc) 3. Define the application URLs. ![AS2 profile configuration step 3](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-config-as2-3.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=ba3f3dda945c0f334bab8593fbb1b47a) To edit an existing profile, click the ellipses on a card and choose **Edit**. Once a profile has been configured, pertinent details are displayed on the profile summary card, including identifiers, URLs, ports, and the number of connectors using the profile. Click the link to the number of connectors to see a list of connector names. From there you can click the link to an individual connector to jump to its location in a [Workspace](../../flows/workspaces). ## Profile Details To view the details of a profile, click the ellipses on its card and choose **Edit**. Each profile has tabs for **Settings**, **Connectors**, and **Logs**. The following image uses the AS2 connector, but the principles are the same for each profile. The **Settings** tab contains all of the profile settings, separated into cards. Each card displays details about the specific settings, and a summary of what they are used for. Click the tablet and pencil icon to edit the values of any card. Use the question mark icon to open the documentation for that setting, or the trash icon to delete that setting's value. ![Completed AS2 profile configuration](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-config-as2-complete.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=87f701057a5244c3ff61dc10e4fc482e) The **Connectors** tab displays a list of all connectors that use the profile, including creation and modification dates, and links to the connector and workspace where it is located. ![Profiles connectors tab](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-connectors.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=906feff592856c494877e46ea74ab8f7) The **Logs** tab displays the list of service logs for the profile. Click the file name link to open details of a log file. The **Status** column indicates the status of the rotating log file. **In Progress** means it is the log that the service is currently writing to. **Complete** means the log is no longer being updated. ![Profiles logs tab](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/profiles-logs.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=69c95c0781bbdfafa45588bdd7d36604) # Reports Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/reports How to create, schedule, and manage reports for data trends in CData Arc. The **Reports** page allows you to create reports about data trends in CData Arc. For example, you can generate a report about file processing times to see which steps in a flow take the longest to complete, or you can use a report to discover which connectors in a flow produce the most errors. ## Availability Reports are only available in the Arc Enterprise package. For information about Arc packages and licensing, see the [Pricing & Packages page](https://arc.cdata.com/order/). Only admin users can create, run, and delete reports. Standard and Support users can view the results of reports that have been run, but they cannot create new reports or run existing reports to generate new results. For more information about user roles, see [User Roles](./settings/user-roles). ## Creating and Managing Reports If you have not created a report, the **Reports** page displays a welcome message and an **Add Report** button. If you have created at least one report, the page lists your reports. ### Creating a Report Follow these steps to create a report: 1. Click **Add Report**. ![Add report button](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/reports_create.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=d6d33282cadf7658f116f3aa5ba9c91f) 2. Enter a **Report Name**. 3. Select the report **Type**. You cannot change this value after the report is created. * **Transactions**—provides data about connectors, including connector Ids, ISA Ids, transactions, and information about processed files. * **Application Usage**—provides data about your Arc license usage. 4. Select an option from the **Presets** list to serve as the template for your report. The template provides a functional base to build your report on, and you can modify the contents of your report as needed. > [!NOTE] > > The **Transactions by ISA Id** and **Transactions by Transaction Set** preset reports rely on header tracking, and they only produce results if header tracking is enabled in the appropriate connectors in the flow. See [X12](../../connectors/x12#advanced-settings) and [EDIFACT](../../connectors/edifact#advanced-settings) for more information about these headers. 5. Click **Add Report** to open the [report view](#report-view) where you can build and run the report. ### Accessing Generated Reports If you have created at least one report, the **Reports** page displays a searchable, sortable list of all of your reports. ![Reports list](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/reports_list.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=e645ddaebe680a8131cd24b8fb96903e) * Click a report in the list to open the [report view](#report-view) for that report. From here you can modify and run the report. * To delete reports, select the checkbox next to each report, click **Remove**, then confirm whether to delete the selected report(s). To delete an individual report, click the ellipses and choose **Remove**. ## Report View When you open a report, the report view consists of two sections: * The main [report view](#main-report-view) on the left shows the results of the most recent report run and includes the **Export** button. * The settings pane on the right consists of the [Builder](#builder-tab) and [Scheduler](#scheduler-tab) tabs. These allow you to build, run, and schedule the report. ### Builder Tab The **Builder** tab allows you to set the report parameters and run it. The parameters are: | Category | Description | | - | - | | Time Period | The period of time that the report covers. Selecting **Custom Date Range** adds two fields for the start date and end date for the report. The default is *All-time*. | | Columns | The information that the report generates. You can include as many columns as you need. See [Columns](#columns) for a list of available options. | | Group by | The fields to use when grouping results in the report. Each selected option receives its own column in the report. See [Group By](#group-by) for a list of available options. | | Filters | Options for narrowing the report results. See [Filters](#filters) for a list of available options. | | Data Summary | Options for summarizing the values of each column. Enabling at least one option creates a **Subtotal** row for each report section and a **Grand Total** row at the bottom of the report. See [Data Summary](#data-summary) for a list of available options. | > [!NOTE] > > Transaction reports all include any [tracked headers](./activity#tracked-headers) you have defined in Arc. They appear in the lists of available column, group by, and filter options, and you can select them just like any other item. #### Columns The following options are available for **Columns**: | Column | Description | Transactions Reports | Application Usage Reports | | - | - | :-: | :-: | | Workspace | Workspace (such as *Default*) | ✓ | ✓ | | Connector Category | Category (such as *Core*, *EDI*, or *MFT*) | | ✓ | | Connector Id | Connector Id (such as *XMLMap1*) | ✓ | ✓ | | Connector Type | Type (such as *XML Map*) | ✓ | ✓ | | Year | Year of the transaction | ✓ | | | Quarter | Quarter of the transaction | ✓ | | | Month | Month of the transaction | ✓ | | | Week | Week of the transaction | ✓ | | | Sender ISA Id | The ISA Id of the sender of an EDI document | ✓ | | | Receiver ISA Id | The ISA Id of the recipient of an EDI document | ✓ | | | Transaction Set | The standardized identification code for the type of document transmitted | ✓ | | | All Transactions | Total number of sent and received transactions | ✓ | | | Transactions Sent | Number of transactions sent | ✓ | | | Transactions Received | Number of transactions received | ✓ | | | Total Processing Time | Processing time (in milliseconds) of all transactions | ✓ | | | Total File Size | Size of all files sent and received | ✓ | | | Successful | Number of successful transactions | ✓ | | | Error | Number of transaction errors | ✓ | | | Warning | Number of transaction warnings | ✓ | | #### Group By The following options are available for **Group by**: | Group by | Transactions Reports | Application Usage Reports | | - | :-: | :-: | | Workspace | ✓ | ✓ | | Connector Category | | ✓ | | Connector Id | ✓ | | | Connector Type | ✓ | ✓ | | Year | ✓ | | | Quarter | ✓ | | | Month | ✓ | | | Week | ✓ | | | Sender ISA Id | ✓ | | | Receiver ISA Id | ✓ | | | Transaction Set | ✓ | | #### Filters The following options are available for **Filters**: | Filters | Transactions Reports | Application Usage Reports | | - | :-: | :-: | | Workspace | ✓ | ✓ | | Connector Category | | ✓ | | Connector Id | ✓ | ✓ | | Connector Type | ✓ | ✓ | | Status | ✓ | | | Sender ISA Id | ✓ | | | Receiver ISA Id | ✓ | | | Transaction Set | ✓ | | #### Data Summary The following options are available for **Data Summary**: | Summary | Transactions Reports | Application Usage Reports | | - | :-: | :-: | | Total | ✓ | ✓ | | Average | ✓ | | | Minimum | ✓ | | | Maximum | ✓ | | ### Scheduler Tab The **Scheduler** tab allows you to set an automatic schedule for running the current report. This can be useful if you have reports you are interested in viewing regularly, without having to log into the application to get the data. By default, the **Frequency** is set to *None*, which means that the report does not run unless you manually execute it. To set the report to run automatically, set the **Frequency** to *Monthly*, *Weekly*, or *Daily* and configure the settings as needed. After configuring the schedule, click **Save** or **Apply** at the bottom of the **Scheduler** tab to save your changes. You can also click the **Revert Changes** icon to revert all changes to the last saved schedule values. > [!NOTE] > > If you have a schedule set, you can still manually run the report at any time by clicking **Run** on the **Builder** tab. #### Email Report If you configure a schedule and select **Email Report**, the application automatically sends an email with the report results when the scheduler runs. When this feature is enabled, two fields appear: * **Subject** specifies the subject of the automated report email. * **Recipients** accepts a comma-separated list of email addresses for the report recipients. The email feature uses the same email server configuration that is specified on the [Alerts](./settings/settings-page#alerts) page. ### Main Report View The main report view displays the results generated the last time the current report was run. You can sort each column of the report in ascending or descending order. ![Main report view](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/reports_mainview.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=7ccf7129b7c5ef9453f1c1c2ff1244de) > [!NOTE] > > When you set a [Group by](#group-by) column in your report, the report results are automatically sorted by that column in descending order, as shown in the following image. If you try to sort by other columns in the table, nothing happens because the Group By always takes precedence. ![Report grouped by column](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/reports_groupby.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=12b9ea950fb4ca54d199383d8e35dcb9) Above the report results, you can rename the report by clicking the tablet and pencil ![tablet and pencil](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_expressioneditor_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=ca9752121351859d881672fcb4d70060) icon next to the report name. * Click **Export** to export the most recent report results to a CSV file. * Click **Save** to save all changes to the current report. ### Run, Save and Revert At the bottom of the right-hand pane you always have three options: * Click the **Revert Changes** ![revert changes](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/reports_revert_icon.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=fef39241fc65bd497ed20ace41720886) icon to revert all changes to the last saved version of the report. * Click **Save** to **Save Changes** to the existing report or to **Save as a New Report**. * Click **Run** to run the report with the current settings and display the results in the main report view area. # Security Best Practices Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/security-best-practices Security-related features and recommended best practices for CData Arc installations. This page provides information on security-related features available in CData Arc and describes the best practices we recommend you follow. Security topics are divided broadly into two categories: * Securing the infrastructure upon which Arc operates * Securing data in Arc Both of these sections describe relevant security concepts and contain subsections with our best practice recommendations. ## Securing the CData Arc Infrastructure Many security settings are managed by the server configuration files. When you use the embedded web server included with Arc, the server configuration files are found in the following locations: * .NET Edition—the `Web.Config` file in the `www` folder of the Arc installation directory (by default, `C:\Program Files\CData\CData Arc`) * Java Edition—the `arc.properties` file in the Arc installation directory When deploying Arc to an external server, you should use the configuration file for that external server instead. ### Enabling TLS/SSL for Inbound Traffic CData strongly recommends that you enable TLS/SSL on your server. TLS/SSL uses public-key/private-key cryptography to encrypt communication channels between a client and server. This encryption protects the confidentiality, authenticity, and integrity of data in Arc. In addition, through the use of digital certificates, TLS offers facilities for client/server identity confirmation. Enabling inbound TLS/SSL can be divided into two categories: * TLS for the host web server, which relates to HTTP traffic, including protocols (such as [AS2](../../connectors/as2) and [AS4](../../connectors/as4)) that use HTTP underneath. Since the web server that hosts Arc is used to handle these requests, TLS must be enabled at the web server level. * TLS for all other servers, which relates to non-HTTP traffic, such as [FTP](../../connectors/ftp) and [OFTP](../../connectors/oftp). Since the servers handling these inbound connections are not the same as the web server hosting Arc, you must enable these options for the appropriate protocol in the Arc interface on the tabs on the [Profiles](./profiles) page. #### TLS/SSL for the Web Server * For the .NET edition of Arc, see either [Using the Embedded Server](../installation/windows-edition#using-the-embedded-server) or [Using IIS](../installation/windows-edition#using-iis). * For the Cross-Platform edition, see [Enabling TLS/SSL](../installation/cross-platform-edition#enabling-tls/ssl). #### TLS/SSL for Other Servers Use the [Profiles](./profiles) page to enable TLS/SSL for the following servers: * [FTP server](../../connectors/ftp-server) * [OFTP server](../../connectors/oftp) * [HL7 MLLP server](../../connectors/hl7mllp) Each server has a tab on the Profiles page where you can enable or disable TLS/SSL. ### Configuring Firewalls and Using a DMZ In order for Arc to send and receive messages, it needs to be free from firewall interference on the relevant sending and receiving ports. The best option is to whitelist specific IPs within the firewall to ensure that traffic only comes from expected sources. In cases where this is not possible, the firewall must be opened on critical ports where Arc listens for inbound messages. Arc protects configuration resources behind a login portal. By default, the Arc web server listens on port 8001. You can configure ports for non-web protocols directly in the application. For example, you can configure the [OFTP server](../../connectors/oftp) listening port in the OFTP server tab on the Profiles page. #### DMZ The .NET edition of Arc includes support for establishing a reverse SSH tunnel to host Arc in a demilitarized zone (DMZ) to avoid opening the firewall on a private network directly. See [DMZ Gateway](./dmz-gateway) for more information on cloud hosting. ### Separating Public Endpoints from the Web Interface By default, Arc uses the same web server port to host both the admin console (where Arc is configured and managed) and the public receiving endpoints that remote partners use to send data to the application. The admin console is protected by credentials to ensure that remote partners cannot gain access. However, hosting the public receiving endpoints on a separate port ensures that network/firewall rules can be used to add a layer of redundancy and enhance security. See [Split-Plane Deployment](./split-plane-deployment) for more information on hosting public receiving endpoints on a separate network port. ### Hardening Server Security Settings You can increase the security of your Arc server by setting the `ScriptingEngineHardeningLevel` configuration parameter in the Arc configuration files. By default, this parameter is set to `0`, which does not block any operations. You can set it to a value between `0` and `3`, where each higher value increases security by disabling certain operations. The following table shows which operations are disabled at each hardening level: | Level | Disabled Operations | | - | - | | 0 | All operations are allowed. This is the default. | | 1 | The most vulnerable operations are disabled. This includes any operations that allow the user to interact with the host system in an arbitrary way. | | 2 | In addition to the operations disallowed in level 1, this level disables operations for creating new users, accessing files on disk in a non-arbitrary way, and introducing recurring tasks. | | 3 | In addition to the operations disallowed in levels 1 and 2, this level disables operations for obtaining sensitive information from the server or performing arbitrary http requests. | Use the following sections to set the hardening level for your installation. #### .NET Use a text editor to open the `Web.Config` file in the `www` folder of the installation directory. At the bottom, inside the `` tag, add the following line and set `value` to your desired level: ```xml theme={null} ``` #### Java Use a text editor to open the `arc.properties` file in the installation directory. Add a line for the following property, and set the number to your desired level: ``` cdata.initParameters=ScriptingEngineHardeningLevel:0 ``` ### Interaction with External Malware and Antivirus File Scanners Arc expects to have exclusive access to the contents of the Application Directory for the duration of file processing. The application uses time-sensitive resources such as resource locks in the application directory to ensure that multiple threads do not process the same files. It scans files in the **Transactions** tabs of connectors to apply metadata to messages, and when it moves files between connectors it immediately moves files between connector directories to ensure rapid processing across multiple steps in a flow. Processes that scan files and file handles can impact the performance in Arc and cause unexpected behaviors in the application, including interruptions in file automation, failure to log traffic, failure to move messages between connectors, degradation of processing performance, and so on. CData recommends that you do not point file scanning utilities to the Arc Application Directory or any of its subdirectories. If you need to perform virus or malware scanning on files that arrive on the local system, we recommend that you use the [File connector](../../connectors/file) to move files to a path on a disk outside of the Application Directory. That location can then be used as a staging folder for file scanning. #### Use the File Connector to Continue Processing Scanned Files When you receive files from a folder on disk that is shared with another process, you can use the [File connector](../../connectors/file) to set a **Receiving Delay** to configure the time (in seconds) that the connector waits before receiving and processing incoming files. The default interval is 5 seconds. Increase that value to allow more time for the file scanner to complete. ## Securing Data in CData Arc The following sections relate to the security of data in Arc, including the messages and data that flow through the application and the [flow](../../flows) configurations themselves. ### Enabling TLS/SSL for Outbound Traffic Arc automatically parses whether to negotiate TLS/SSL based on the target URL of outbound connections. For example, if an outbound [AS2](../../connectors/as2) connection is targeting an `https` URL, Arc negotiates TLS. TLS/SSL is strongly recommended for transferring data over protocols that do not include built-in encryption mechanisms. This means that web traffic should be sent to `https` endpoints, FTP traffic should be directed to `ftps` endpoints, and so on. Other than correctly configuring the target URL, no additional steps are necessary to ensure Arc negotiates TLS/SSL encryption. ### Validating TLS/SSL Server Certificates and SSH Keys When making an outbound connection to a TLS/SSL or SSH server, the connector establishing the connection (for example, [REST](../../connectors/rest), [AS2](../../connectors/as2), or [SFTP](../../connectors/sftp)) attempts to validate the certificate or key provided by the server. If a server certificate/key is not configured in the connector settings, Arc uses the validation process present in the underlying system (the OS or JRE) to determine if the server's certificate is trusted. You can configure connectors with a specific TLS/SSL certificate or SSH key thumbprint to override the underlying system behavior. When making an outbound connection, if the server's certificate matches the certificate configured in the connector, the connector trusts the server. This option provides the highest level of security. Connectors implicitly trust the server's identity when you set the **TLS Server Certificate** field to *Any Certificate*. You should only do this when there is no concern about server impersonation attacks. ### Managing Users and Admins and Accessing the API Arc maintains a set of username and password credentials for users who are authorized to log in to the Arc web interface and use the application. For more information on users and roles, including the Admin role, see [Users](./settings/users) and [User Roles](./settings/user-roles). Users with the Admin role are allowed to access and change passwords of other users. The credentials for the Admin user created when Arc is first launched should be securely stored in an external location to ensure that this account is both secure and recoverable. ### Admin API Users can perform all application management tasks via calls to the Admin API, so securing access to this API is just as important as securing access to the web interface. When an Arc user is created, you can generate an auth token that allows this user access to the API (with restrictions according to that user's role in the application). For more information on using an auth token to access the Admin API, see [API Authentication](../../admin-api/admin-api-authentication). > [!NOTE] > > A user's auth token is not displayed after it is created, so it should be stored securely in an external location. ### FIPS Compliance The Federal Information Processing Standards (FIPS) is a set of security standards developed by the United States federal government for use in computer systems to ensure security, interoperability, and data integrity. To enable FIPS in Arc, follow these edition-specific instructions. #### Java Edition To enable FIPS in the embedded Jetty web server, you must first download and install the required Bouncy Castle FIPS jar files. Download the following jar files from [[https://www.bouncycastle.org/download/bouncy-castle-java-fips/#latest](https://www.bouncycastle.org/download/bouncy-castle-java-fips/#latest)](https://www.bouncycastle.org/download/bouncy-castle-java-fips/#latest) and place them in the `lib` folder next to `arc.jar`: * `bcutil-fips-2.x.jar` * `bc-fips-2.x.jar` * `bcpkix-fips-2.x.jar` * `bctls-fips-2.x.jar` The configuration of FIPS depends on whether the embedded Jetty web server is configured to use HTTP or TLS/SSL: ##### HTTP If Arc is running behind a reverse proxy that provides TLS/SSL to clients, and the server running Arc communicates over HTTP with that proxy, simply add `cdata.fips.enabled=true` to your `arc.properties` file. ##### TLS/SSL To enable FIPS when configured to use TLS/SSL (HTTPS), you must first create a BCFKS keystore using the following command: ``` keytool -genkeypair -alias jetty -keyalg RSA -keystore keystore.bcfks -storetype BCFKS -storepass xA123456 -keypass xA123456 -dname "CN=localhost" -validity 365 -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providername BCFIPS -J-cp -J"bc-fips-2.1.0.jar" ``` Once you have created your keystore, update your `arc.properties` file with the following configuration: ``` cdata.fips.enabled=true ;; TLS cdata.tls.port=8443 cdata.tls.keyStoreType=BCFKS cdata.tls.keyStorePath=/pathto/keystore.bcfks cdata.tls.keyStorePassword=xA123456 ``` > [!NOTE] > > When enabling FIPS in the embedded Jetty web server, the minimum password length is 14 characters. You cannot log in to Arc if your password is less than 14 characters when FIPS is enabled. #### .NET Edition To enable FIPS in the embedded web server for .NET or when hosting the application in IIS, add the following statement to your `Web.Config` file. ```xml theme={null} ``` # Additional Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/additional-settings Miscellaneous configuration options for CData Arc including connector visibility, logging, and time format settings. The following options appear in the **Additional Settings** portion of the **Advanced** page. | Setting | Description | | - | - | | **Beta Connectors** | When this is enabled, beta connectors are shown in the [Connectors](../../../connectors) list on the [Flows](../../../flows) page. | | **Deprecated Connectors** | When enabled, deprecated connectors are also shown in the list of connectors. | | **Local IP** | The local interface to bind to for Send connections. | | **Base URL** | By default, URLs to public endpoints within the application are automatically generated based on the context of the request the application received to show the web page. This setting is used to generate URLs, which can be useful in environments where the application is running behind a proxy server. | | **Log Level** | The verbosity of the logs written to the [application logs](../activity#application-logs). This setting does not affect the [automation logs](../activity#automation-logs), which follow each connector's own **Log Level**. | | **Tracked Headers** | A static list of [headers](../activity#tracked-headers) that should be tracked whenever they are added to messages. | | **Message Buffer Size** | Size of the message buffer (in KB). | | **Time Format** | Whether to display time in a 12 or 24 hour format. | | **Other Settings** | A semicolon-separated list of hidden configuration settings in the form `setting1=value1;setting2=value2`. This should only be necessary in special circumstances. | # Admin API Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/admin-api-settings How to configure CORS and authtoken settings for browser-based Admin API access in CData Arc. The **Admin API Options** portion of the **Security** page covers two types of Arc configuration options. ![Admin API settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_admin_api.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=dd96f230dbd3c37c68e0a5b6d7ce4703) ## Cross-Origin Resource Sharing (CORS) CORS allows browser-based clients to connect to Arc. Without CORS, browser-based scripts cannot connect to Arc because of the same-origin policy enforced by the browser. This policy restricts client-side scripts and documents from loading resources outside their origin. The origin of a script consists of the protocol, host, and port. The following table describes the available CORS settings. | Setting | Description | | - | - | | **Enable cross-origin resource sharing (CORS)** | Whether or not CORS is enabled. The rest of these options are only available when you check this box. | | **Allow all domains without '\*'** | If enabled, domain origins are not restricted to a specific list. | | **Access-Control-Allow-Origin** | The comma-delimited list of domain origins to allow. Included as an HTTP response header. | | **Access-Control-Allow-Credentials** | Whether or not user credentials such as cookies are allowed in cross-origin requests. Included as an HTTP response header. | | **Access-Control-Allow-Methods** | The comma-delimited list of methods to allow. Included as an HTTP response header. | | **Access-Control-Allow-Headers** | The comma-delimited list of headers to allow. Included as an HTTP response header. | | **Access-Control-Max-Age** | The maximum duration (in seconds) that Access-Control response header values can be cached. | ## Authtoken in URL Check this to allow the connector to pass the auth token in query string parameters for authentication. For example: `https://myarcinstance.com/api.rsc/connectors?@authtoken=myAuthTokenValue`. # Advanced Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings Overview of advanced configuration options in CData Arc, including cleanup, performance, AI, migration, and additional settings. The **Advanced** tab holds several categories of configuration options, each of which has its own documentation topic: * [Cleanup Settings](./cleanup-settings) cover options related to log file cleanup intervals. These options are designed to maximize application performance. * [Performance Settings](./performance-settings) options allow you to define global settings for the [Automation](../../architecture#automation) service, and to analyze worker pool performance. * [AI Settings](./ai-settings) options control the behavior of the AI features. * [Migration Settings](./migration) options enable you to perform a full import or export of your application, including all of the global application data and settings. * [Git Version Control](./git-version-control-settings) enables you to set up a Git repository to aid in configuration management. * [Additional Settings](./additional-settings) cover a variety of additional configuration options. # AI Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/ai-settings How to configure AI provider settings for AI features in CData Arc. This section contains the configuration options that control the behavior of Arc's in-product AI features. To connect an external AI client to this Arc instance, see [Remote MCP Server](../../../admin-api/remote-mcp-server). You can also use the configured provider to [diagnose transaction errors](../activity#diagnosing-transaction-errors) from the **Activity** page. Click **Configure** to open the **Configure AI Settings** pane. ![Configure AI settings pane](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_ai.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=695e3ea13471a26f37514364d6a6b3ed) | Setting | Description | | - | - | | **AI Provider** | Choose your AI provider from the list. The supported providers are Claude, DeepSeek, Gemini, Grok, Ollama (cloud and local), OpenAI, and OpenAI Compatible. The following settings vary based on your choice of provider. | | **API Key** | Enter your API key to authenticate requests (all providers except Ollama local). | | **Base URL** | Specify the endpoint URL where your *Ollama (Local)* or *OpenAI Compatible* service is running (for example, `http://localhost:11434`). | | **Model** | Choose the specific language model to use. Available models depend on your provider. | > [!NOTE] > > Not all open source models support the ability to set a JSON schema to use for the response that Arc requires. Ensure the **Model** you choose has that capability. An AI provider is also required to analyze the worker pool with the **Analyze Worker Pool** button in [Performance Settings](./performance-settings#analyze-worker-pool). The button is disabled until you configure a provider. # Alerts Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/alerts How to configure monitoring alerts for system health, certificate expirations, processing errors, and critical events in CData Arc. Use the **Alerts** tab to configure intelligent monitoring that keeps you informed with timely alerts for system health, certificate expirations, processing errors, and critical events before they impact your operations. Click **Configure** in the **Email Settings** portion of the page to enable email alerts or status reports. Complete the **Email Server Settings** fields to configure the email recipients, subject lines, and SMTP authentication options. ![Email server settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_alerts_emailserver_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=f141327d4576e26c56d66bce9b6dd131) **System Health Alerts** enables you to specify system-related alerting options, including alert intervals, report intervals, and several system threshold settings. ![System health settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_alerts_system_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=cf8b159d3c36e630c09569e1a65059f3) Click the **Last report** link to see data related to memory and disk space usage, and the application's current transaction count. ![System health report](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_alerts_system_health_report_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=18002f0746d3c273a8888c64e827c933) **Syslog Settings** contains options for sending application logs to an external Syslog server. If you enable this option, use the fields to specify the remote host connection settings. ![Syslog settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_alerts_syslog_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=e9c2bfff49156745a264d77277b32213) **General Alerts** enables you to toggle alerting options, including sending emails about errors, certificate expirations, and status reports, and enforcing expected volume SLAs for connectors. If you use the .NET edition, you can also choose to log error messages to the Windows Application Event log. ![General settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_alerts_general_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=36bd7b40f4e5d493e4de5763799fbdc4) To make changes to any of the alert settings, click the tablet and pencil ![tablet and pencil](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_expressioneditor_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=ca9752121351859d881672fcb4d70060) icon. ### Custom Alerts You can send a custom alert based on a specific condition when a message is processing. To do this, add the [Notify](../../../connectors/notify) connector to your [flow](../../../flows) and configure it to notify you when a message is received by that connector. # Application Database Migration Utility Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/app-db-migration How to move your Arc application data from the embedded database to an external database. The application database migration utility enables admin users to move the CData Arc backend application data from the embedded database to an external database. The migration engine includes a source-database router that auto-detects the embedded database, then handles the migration. The embedded database varies by platform: * Cross-Platform: H2 * .NET: SQLite Embedded database limitations such as performance at scale and limits on concurrent access are common reasons why you might want to use this utility. The following external databases are supported targets: * SQL Server * MySQL * PostgreSQL The migration moves all tables from the backend database to the target database, including tables such as the following: * Transaction logs * Message logs * Audit logs * Connector state * SLA status data * General application data Before you begin, consider the timing of your migration, because the mandatory Safe Mode blocks all file processing. You might want to run the process during a planned maintenance window. ## Migrate the Application Database To launch the utility, navigate to **Settings > Advanced**, scroll to the **Migration** section and select **Application Database Migration**. 1. Application database migration is not supported in high availability (HA) deployments. You must stop any other Arc nodes and acknowledge that all other nodes are stopped before you can continue. Check **I understand all other HA nodes must be stopped**, then click **Next**. 2. You must be in [Safe Mode](#safe-mode-behavior) to continue the migration process. This disables all background services, stops all connectors, and blocks inbound endpoints to ensure data integrity throughout the migration process. Click **Enter Safe Mode**. 3. Click **Select Database**, choose your target database type from the list above, then provide the connection details for the target database. 4. Click **Test Connection** to validate the connection. 5. Click **Begin Migration**. Data is migrated table by table. Migration time is based on data volume and can take anywhere from a few minutes to several hours. 6. Once the migration completes successfully, you are given a connection string. 7. Add the connection string to your configuration. The steps depend on your platform: * **Cross-Platform:** Add the string to your configuration (see [Configure the Cross-Platform Application Database](../../installation/cross-platform-edition#configure-the-application-database)). After you update the configuration to point to the new database, restart Arc. * **.NET:** When the migration completes, Arc remains in Safe Mode until the new connection string is applied and Arc restarts. Background services stay stopped, and files and transactions are not processed. Click **Copy** to copy the connection string, then update the `AppDb` entry in `Web.Config`: 1. Open `Web.Config` and locate the `` section (be sure to uncomment the section). 2. Add or replace the `AppDb` entry with the following, where `` is the connection string you copied: ```xml theme={null} ``` This example uses PostgreSQL. For SQL Server, set `providerName` to `System.Data.CData.SQL`. For MySQL, set it to `System.Data.CData.MySQL`. For more information, see [Configure the .NET Application Database](../../installation/windows-edition#configuring-the-application-database). 3. Save `Web.Config`. Arc restarts automatically, so you do not need to restart it manually. ## Safe Mode Behavior When the migration begins, Arc enters *Safe Mode*, which affects the entire application by putting it into a state where nothing can be written to the current database: * All connectors are stopped, which means no send or receive file processing. * Background services are paused, including automation, SLA monitoring, connector monitoring, scheduled reports, disk space monitoring, and so on. * Inbound endpoints are blocked, which means that API requests, webhooks, Flow API calls, SFTP server connections, and email inbox all return errors. * The UI is restricted: all logged-in users see a **Safe Mode Active** message, which prevents them from performing normal operations until the migration is complete. * Admin users have access to the **Start**, **Abort**, and **Monitor Progress** migration controls. * Non-admin users see the Safe Mode status but cannot manage the migration in any way. Connection string details are redacted. ## Recovery and Resilience * Crash recovery: If the Arc process terminates unexpectedly during migration, the migration state is persisted in the backend database. On restart, Arc detects the incomplete migration so you can resume from where it left off instead of starting over. * Background services remain paused during recovery until you explicitly complete or cancel the migration. # Certificates Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/certificates How to manage TLS/SSL certificates in CData Arc. The **Certificates** tab displays a sortable table of all certificates that are used and managed by the application. Use this tab as a reference to quickly determine which certificates are in use by which connectors in your flows. ![Certificates table](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_certificates.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=b1e98c0a501c6021cba30282fdb520a9) If you have the appropriate permissions, you can add or upload new certificates, edit the names of existing certificates, download certificates, and remove those you no longer need. Click **Add Certificate** to see the options for [creating](#creating-certificates) or uploading certificates. To edit, download or delete a certificate, click the ellipses that appear when you hover over a certificate and choose one of the options. Use the search bar to search in columns such as **Name** or **Serial** to find specific certificates. Hover over a number in the **Associated Resources** column to see the connector Ids of all connectors using a specific certificate. Click a connector Id to go directly to its settings. All certificates created or uploaded in CData Arc are stored in the `data` directory. Typically, certificates come in public/private pairs with the same filename but different file extensions: `.pfx` for private and `.cer` for public. Certificates in other formats that are uploaded to the application are stored as-is. Arc enumerates the certificates in the `data` directory to determine the options available when configuring a certificate in the UI. Adding a certificate file to the `data` directory accomplishes the same thing as uploading a certificate in the UI. ## Creating Certificates To manually create a new certificate, choose **Create Certificate** under **Add Certificate**: ![Add a new certificate](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/certificate_add.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=b4d8403a00b3b21e2855eb013262f378) The following fields are required: | Setting | Description | | - | - | | **Common Name** | The hostname of the server using the certificate. It is used in conjunction with the serial number to identify the certificate. | | **File Name** | The name of the certificate file, with a `.pfx` extension. The corresponding public key certificate is given the same name with a `.cer` extension. | | **Serial Number** | A unique serial number that is used in conjunction with the common name to identify the certificate. | | **Password** | The password required to access the private key. | | **Validity Period** | Determines the expiration date of the certificate. | | **Key Algorithm** | The type of key algorithm to use. Options are: RSA (the default), ED25519, and X25519. | | **Key Size** | (RSA only) Whether to create a 512, 1024, 2048, or 4096-bit key. | | **Public Key Type** | (RSA only) Whether to create an X.509, OpenSSH, or SSH2 public key. | | **Signature Algorithm** | (RSA only) The algorithm to use when applying a digital signature to the certificate to verify its authenticity. | | **Create Paired Certificate** | (ED25519 only) Whether to also create a paired X25519 key exchange certificate. | | **Signing Certificate** | (X25519 only) The ED25519 certificate used to sign outgoing messages. | The remaining fields are optional, but you can use them to add further context and metadata to the certificate. When you are finished, click **Add Certificate**. # Cleanup Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/cleanup-settings How to configure log file cleanup intervals and archiving options in CData Arc. Arc automatically cleans up log files at regular intervals to maximize application performance. Use the **Cleanup Settings** section on the **Advanced** page to specify whether to **Archive** or **Delete** the log files, set the **Interval** at which to delete them, and specify the **Archive Type** and location for archived files. You can also manually start a cleanup task from this page. ![Cleanup settings page](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_cleanup.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=bc15301e6fb713335c5f3f4dfcdcb417) | Setting | Description | | - | - | | **Type** | Specify whether to archive or delete logs during cleanup. | | **Interval** | Specify (in days) the cleanup threshold. For example, when set to 5 days, only logs and database entries over 5 days old are cleaned up. [Automation logs](../activity#automation-logs) use the shorter of seven days and this interval. | | **Include files in Sent folders** | Archive or delete files in the connector's Logs/Sent folder that have been successfully sent in the flow. Sent files can consume large amounts of disk space. This option frees up disk space by cleaning up sent files when the cleanup runs. | | **Include files in Send folders** | Archive or delete files that accumulate in the connector Send folders. | | **Include files in Receive Folders** | Include files in Receive folders in cleanup operations. | | **Archive Type** | Specify whether the archive should be sent to your local file disk or to an S3 bucket. Only applicable when **Type** is set to **Archive**. | | **Local Archive Folder** | Applicable when the **Archive Type** is **Local Disk**. By default, the archive folder is a folder called `archive` in your application directory (`archive` is the only relative path supported). To use a different location, enter an absolute path. If you leave the field empty, each [connector folder](../../architecture#connector-folders) contains its own **Archive** folder. | | **S3 Options** | If the **Archive Type** is **S3**, set the following S3-specific options: **S3 Bucket**, **S3 Access Key**, **S3 Secret Key**, and **S3 Region**. | | **Run Cleanup** | Manually starts a cleanup task. A global cleanup also removes [automation logs](../activity#automation-logs) that exceed the retention rules. A workspace-limited cleanup does not remove them. | > [!NOTE] > > If Arc alerts you that the number of transactions in your application database has exceeded the threshold for that database, changing the cleanup settings on this page is one way to reduce the number of stored records and bring the count back below the threshold. Your other option is to migrate to a database with a higher threshold. For threshold values by database, see the application database configuration section for your implementation: [Cross-Platform edition](../../installation/cross-platform-edition#configure-the-application-database) or [.NET edition](../../installation/windows-edition#configuring-the-application-database). ## Include files in Send folders When a large number of files accumulate in connector Send folders, the performance of the application can suffer. These files can accumulate for various reasons, including: * A connector exceeds the maximum number of retries for a message and no error path is configured for that connector. * The Send folder for a connector was overridden in a legacy version of Arc and pointed at a shared folder that the application could not clean up. The ability to change the Send folder and apply Send Filters is deprecated in current versions of Arc. You should use error paths instead. > [!NOTE] > > If you enable this feature, ensure that your connectors process files in the Send folder on a shorter interval than the cleanup interval you specify on this page. Otherwise, the application might clean up the files in the Send folder before the connectors can process them. # Connections Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/connections How to configure shared connection parameters for use across multiple connectors in CData Arc. The **Connections** tab lists configured connection parameters that can be shared among multiple CData Arc connectors. This allows for centralized connection management and simplified configuration in individual connectors that connect to the same source using the same credentials. ![Connections settings tab](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_connections.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=01d1948203ef56aa20ea91da24eacc03) It is useful to understand the difference between CData drivers, API Profiles, and Arc connectors. The *drivers* provide the underlying connectivity to external systems, *API Profiles* extend the drivers to add codeless connectivity for REST-based APIs and web services, and *connectors* build on the drivers and API Profiles to handle data exchange, transformation, and routing within Arc. You configure the shared driver- and API Profile-level connection details on the **Connections** tab, which connectors then reference so the same connection can be reused consistently across multiple flows. * Find documentation for drivers at [[https://www.cdata.com/kb/help/](https://www.cdata.com/kb/help/)](https://www.cdata.com/kb/help/). Find the driver you want in the **Drivers & Data Connectors** section of the page, then click the **Online Documentation** link for the **ADO.NET** or **JDBC** editions. This enables you to find information on how to use the CData driver-based connectors. * Find documentation for API Profiles at [[https://www.cdata.com/apidriver/#profiles](https://www.cdata.com/apidriver/#profiles)](https://www.cdata.com/apidriver/#profiles). Find the API Profile you want in the **Pre-configured API Profiles** section of the page, then click the **Docs** link near the top of the page. ## Add a Connection To create a new shared connection, click **Add Connection** and select the connection target. Arc ships CData drivers for some connectors with the application, but not all. On the **Select Connector** pane, if there is a download button on the connector you need (the CData Act-On connector in the following image has one), use it to download and install the driver or API Profile, which enables the connector for use. ![Select connector pane with download button](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/connection_select_new.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=bbd5b34bfe2b627b310034d7ae888c6a) After you click the download button, the **Install Connector** pane opens and specifies the version number. At this point, it asks whether you want to install manually or automatically. Typically you want to install automatically. However, if CData Support has provided you with a `.zip` file, you can select manual, then use **Choose File** to supply the `.zip` file from them. ![Manual connector installation pane](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/connection_manual_install_driver.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=bf292f65e23ac9c04d2e0e8a4de72210) > [!NOTE] > > * If you are installing to the [.NET edition](../../installation/windows-edition) and encounter a permission error during driver installation, grant `write` permission to `www\Web.Config` and `www\bin` for the Arc service account (Network Service by default). > * If you are installing to the .NET edition, you are warned that the application must restart. Once the installation is complete, or if you did not need to download and install the driver or API Profile, you still need to provide the specific connection parameters. These vary by target, and might require OAuth redirection. Click **Add and Test** to save the connection. If the test fails, the connection is still saved, but you can continue to edit it to resolve any errors. Once the shared connection is created, you can select it from the **Connection** dropdown in connectors that support shared connections. ## Update Versions Administrators can also use the **Connections** tab to update versions. You can update to a new version when one is available, or select a version that meets your needs. Do this by: * Clicking **Add Connection** * Editing an already configured connection by clicking the link to the connection name When you use the **Add Connection** option, click the download button. This opens the **Update Connector** pane. ![Update connector pane via download](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/connection_update_by_download.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=506c449b1ecb3c4422755f4489790806) Choose the type of update you want from **Update Type**, then click **Update**. The options are: * Automatically update connector. * Manually update connector: If you use this option, use the **File** field to upload the `.zip` file to install. * Select from available versions (only appears if the currently installed version is not the latest, or if multiple versions are available): If you use this option, use the **Update Version** field to select the version to install. To edit an existing connection, click the link to the connection name. Click **Update** to update the version. The **Update Connector** pane opens with the same set of update options described above. ![Update connector pane via edit](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/connection_update_by_edit_263.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=c314872e3967da4ce223333051f21a01) # Encryption at Rest Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest How CData Arc encrypts message data at rest with AES-256, including the EncryptionAtRestMode setting, its modes, and operational considerations. Arc supports application-level encryption at rest for [message data](../activity#messages-tab). When enabled, all data files stored in Arc's internal storage directories are automatically encrypted using AES-256-GCM with streaming, providing cryptographic protection for data at rest without requiring changes to existing workflows. Encryption at rest addresses data-protection requirements for standards such as PCI DSS, HIPAA, GDPR, and SOX. Encryption applies to message data in all of Arc's internal storage locations (the Send, Receive, Logs, Pending, and temporary directories) for all connectors. > [!WARNING] > > Message data encrypted at rest can only be decrypted with your master key. If you configure a [custom master key](../../installation/custom-master-key) and lose access to it, encrypted message data **cannot be recovered**. Implement secure key backup and recovery procedures before enabling encryption at rest. The following objects are **not** encrypted by this feature: * Configuration files: sensitive configuration values are encrypted separately by Arc's existing mechanism. * The application database Encryption is configured in the **Additional Settings** section of **Settings > Security**. It is disabled by default. To enable it, select one of the following modes: | Mode | Write behavior | Read behavior | | - | - | - | | **Disabled** | Plaintext only | Plaintext only | | **Compatibility** | Writes plaintext files | Reads encrypted and plaintext files | | **Standard** | Writes encrypted files | Reads encrypted and plaintext files | | **Strict** | Writes encrypted files | Reads encrypted files only; plaintext files cause an error | * **Disabled** is the default. Encryption must be explicitly enabled. * **Standard** is recommended for most users enabling encryption: it encrypts new files while still accepting existing plaintext files. * **Compatibility** writes plaintext but reads both encrypted and plaintext files. It is a bridge mode for safely moving between other modes (see [Switching Modes in a Clustered Environment](#switching-modes-in-a-clustered-environment)). It never encrypts on its own: to start encrypting, use **Standard**. * **Strict** is intended for high-security or compliance environments where plaintext files must never be accepted. > [!NOTE] > > Restart Arc for encryption mode changes to take effect. ## Migrating Existing Data Enabling encryption does not retroactively encrypt files that are already on disk. In **Compatibility** and **Standard** modes, Arc automatically detects whether each file is encrypted and reads existing plaintext files without modification; a file becomes encrypted only when Arc next writes it. Because **Strict** mode rejects any plaintext file, do not switch to **Strict** until no plaintext message files remain in Arc's storage directories. A typical migration path is: 1. Select **Standard** mode so new files are written encrypted while existing plaintext files are still accepted. 2. Allow existing plaintext files to be processed and cleared out of the Send, Receive, Pending, Logs, and temporary directories through normal operation. 3. Once only encrypted files remain, switch to **Strict** mode to enforce encrypted-only reads. In a clustered deployment, follow the steps in [Switching Modes in a Clustered Environment](#switching-modes-in-a-clustered-environment) instead. > [!NOTE] > > Switching to **Strict** mode while plaintext files still exist, or while an external system is still writing plaintext to an FTP/SFTP Server root directory, causes those files to error. See [FTP Server and SFTP Server Considerations](#ftp-server-and-sftp-server-considerations). ## Switching Modes in a Clustered Environment In a [clustered](../clustering) deployment, all nodes share the same configuration and message storage. Because a mode change takes effect only after a restart, you restart nodes one at a time to avoid downtime, and during that rollout the nodes temporarily run different modes. A node that meets a file it cannot read in its current mode fails with a hard error (a **Disabled** node reading an encrypted file, or a **Strict** node reading a plaintext file). To avoid this, change the mode **one step at a time**, following this sequence: **Disabled**, **Compatibility**, **Standard**, **Strict**. Complete a full restart of every node at each step before the next. Do not skip a step. **To enable encryption:** 1. Set **Compatibility** and restart all nodes. Every node can now read encrypted files but still writes plaintext. 2. Set **Standard** and restart all nodes. Nodes begin writing encrypted files, which every node can already read. 3. Wait until no plaintext files remain (see [Migrating Existing Data](#migrating-existing-data)). 4. Set **Strict** and restart all nodes. **To disable encryption:** 1. Set **Standard** and restart all nodes. 2. Set **Compatibility** and restart all nodes. Nodes write plaintext again while still reading the encrypted backlog. 3. Wait until no encrypted files remain. 4. Set **Disabled** and restart all nodes. > [!NOTE] > > On a single node there is no rollout window, so you can switch directly to the target mode. You still cannot select **Strict** while plaintext files remain, or **Disabled** while encrypted files remain. ## Transparent Operation * Encryption and decryption happen automatically; no changes are required to users and existing workflows. * Files are encrypted while stored at rest inside Arc, but are delivered to downstream connectors and trading partners in plaintext. * Encryption uses Arc's existing master key mechanism, so additional key configuration is not required. * Enabling encryption can have an impact on performance, but file-size overhead is negligible. * To use a custom master key, see [Defining a Custom Master Key](../../installation/custom-master-key) for details on how to configure it. ## FTP Server and SFTP Server Considerations Encryption at rest applies to the root directory used by the [FTP Server](../../../connectors/ftp-server) and [SFTP Server](../../../connectors/sftp-server) connectors. * In **Compatibility** or **Standard** mode, external SFTP and FTP clients can still write plaintext files to the Root Directory; Arc accepts and processes them normally. * In **Strict** mode, only encrypted files are accepted; external systems writing plaintext directly to the root directory encounter errors. If you have external systems that write directly to an FTP or SFTP Server root directory, avoid using **Strict** mode. ## Key Management Encryption at rest uses Arc's existing master key, the same key that already protects sensitive setting values, so a separate key setup is not required. See [Custom Master Key](../../installation/custom-master-key) for details on the master key. ### Rotating the Master Key Rotating the master key does not re-encrypt your message data or make existing encrypted files unreadable. Encryption at rest protects files through a key hierarchy: the master key protects an internal data key, which in turn protects each file's encryption key. This means that a rotation re-wraps that internal key rather than rewriting every file. Files encrypted before a rotation remain readable afterward, and no bulk re-encryption step is required. Rotate the master key using the procedure described in [Custom Master Key](../../installation/custom-master-key). # Git Version Control Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/git-version-control-settings How to configure Git-based version control for CData Arc configuration tracking and environment management, including per-workspace repositories. You can use [Git-based version control](../git-version-control) in CData Arc, which enables configuration tracking, audit trails, and environment management. Before you can use it, you must install Git. See [Prerequisites](../git-version-control#prerequisites) for instructions. To complete the configuration of the global (application-level) repository, navigate to **Settings > Advanced > Git Version Control**, then click **Configure**. To version an individual workspace in its own repository, see [Workspace Git Version Control Settings](#workspace-git-version-control-settings). ![Git version control configuration pane](/26.4/images/settings_git_local.png) Select a **Configuration Type**: * **Remote**: connects to a hosted Git repository. * **Local**: creates a standalone Git repository directly on the Arc server. No remote URL or credentials are required. For a remote repository, provide the following: * The URL to your remote Git repository. If you are using SSH, do not enter the HTTPS address here; you must use the SSH address (for example, `git@github.com:username/private-repo.git`). All major Git platforms are supported (such as GitHub, GitLab, Bitbucket, and Azure DevOps). * Your auth scheme. Once you select an auth scheme, the fields required for configuration appear. * HTTPS: The **Username** and **Password** or **Token** to connect to the repository. * SSH: The **SSH Private Key** file and **Passphrase** to connect to the repository. See [Authenticating with SSH](#authenticating-with-ssh) for more information. * OAuth: Click **Get Access Token** to retrieve the access token that allows you to connect. After the repository is initialized, you can still edit the remote URL (see [Edit the Remote URL](../git-version-control#edit-the-remote-url)). You can also switch the **Configuration Type** between **Local** and **Remote** (see [Migrate from a Local to a Remote Repository](../git-version-control#migrate-from-a-local-to-a-remote-repository) and [Convert a Remote Repository to a Local Repository](../git-version-control#convert-a-remote-repository-to-a-local-repository) for details). When you save a change, Arc adds, changes, or removes the Git remote and then applies your credentials. Your local commit history and tracked files are preserved. When you click **Save**, Arc runs an initialization task. This initialization creates a Git repository inside `AppDirectory`, creates a branch named `settings-before-git-init-` that contains the configuration that exists at initialization, and stages those configuration files for commit. This ensures that no configurations are lost in the event that the remote repository already has content that would conflict with those settings. The Arc footer changes to a status bar showing Git metrics and actions such as the current number of local files changed, the current branch, and (for remote repositories) the number of remote commits. For remote repositories, click **Commit and Push** on the toolbar to push the initial commit. For local repositories, click **Commit**. ![Git version control footer toolbar](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/version_control_footer_toolbar.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d59c1b1cf3d5ea75acdc08db5941353f) ## Workspace Git Version Control Settings A [workspace](../../../flows/workspaces) can have its own Git repository, separate from the global repository. To configure it, open the workspace, then click **Settings** in the ellipsis menu. Click the **Git Version Control** tab. See [Global and Workspace Repositories](../git-version-control#global-and-workspace-repositories) for how workspace repositories relate to the global repository. Configure the following: * **Enable Git Version Control**: turns on a separate Git repository for this workspace. If the global repository is set up, a warning explains that enabling version control for the workspace removes it from the global repository, so the workspace is tracked only in the repository that you configure. Shared resources that the workspace uses, such as connections, certificates, and profiles, remain in the global repository. - **Configuration Type**: select **Remote** to connect to a hosted Git repository, or **Local** to use a Git repository on the Arc server without connecting to an external service. * **Repository URL and authentication**: provide the same repository URL, auth scheme, and credentials that the global repository form requires (HTTPS, SSH, or OAuth). See the settings above and [Authenticating with SSH](#authenticating-with-ssh). ![Workspace settings Git Version Control tab with version control enabled](/26.4/images/version_control_workspace_settings_enabled.png) Click **Save** to apply the settings. > [!NOTE] > > Workspace repositories follow a few rules, such as a restriction on the **Default** workspace. See [Global and Workspace Repositories](#global-and-workspace-repositories) for details. > > You can switch the **Configuration Type** between **Local** and **Remote**. > > As with the global repository, you can still edit the remote URL after initialization. ## Authenticating with SSH Follow these steps to authenticate using SSH: * Run `ls -al ~/.ssh` to check for an existing SSH key. * If you have an existing key, you can use it. Alternatively, run this command to generate a new SSH public/private key pair: `ssh-keygen -t ed25519 -C "your_email@example.com"`. * Optionally, add a passphrase to the key for added security. * Use the **SSH Private Key** field to upload your private key. * Add the public SSH key to your Git platform. For more information on how to authenticate with SSH, review your Git platform's documentation. ## Removing a Git Configuration To remove your global Git version control configuration, click the trashcan icon on the settings panel, then choose a **Git Repository Option**: * **Leave the underlying repository intact**: Arc disconnects from the repository, but the `.git` folder remains on disk. You can still interact with the repository directly using Git tools on the server. * **Remove the underlying repository**: Deletes the `.git` folder, fully disabling version control on this installation. Any staged changes are permanently lost. The **Leave...** option is useful when you are using a local repository, because it preserves your commit history and allows you to reconnect later. You can also [convert the repository to a remote repository](../git-version-control#migrate-from-a-local-to-a-remote-repository) in the interface without removing it. After removing the configuration, you can set up version control again by clicking **Configure**. To remove a workspace Git repository, toggle **Enable Git Version Control** off, then click **Save**. You are presented with the same **Git Repository Option** choices as for the global repository, but the results differ for a workspace: * **Leave the underlying repository intact**: Arc disconnects the workspace from its repository, but the `.git` folder remains. Uncommitted changes stay in that repository. The workspace is not tracked by the global repository, even if you have one. * **Remove the underlying repository**: Deletes the workspace's `.git` folder. If you have a global repository, uncommitted changes in the workspace are automatically added to it, and the global repository tracks the workspace again. # Global Settings Vault Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault How to manage reusable values in the CData Arc Vault for use across the admin console. The **Vault** tab enables you to add and manage values that you can reuse throughout the CData Arc admin console. This feature can save time and prevent errors when configuring connectors that reuse certain data. Arc supports external vault providers that let you reference secrets managed outside of Arc, and a built-in local vault. Both types of items appear in the vault table. ## Managing Vault Providers Vault providers determine where Arc resolves vault item values. The local vault is always present and is the only provider you can write to. External providers are read-only and let you surface secrets that already exist in your infrastructure without duplicating them in Arc. To open the provider list, click **Vault Provider Config** on the Vault tab. The **Vault Providers Config** pane opens, listing each configured provider with its **Name** and **Provider** type. The **Local** provider is always the first row and cannot be edited or removed. ![Vault Providers Config pane showing the Local provider and the Add dropdown](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-providers-config.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=594505185c280f8005b0ade08c173d76) ### Adding an Azure Key Vault Provider 1. In the **Vault Providers Config** pane, click **Add > Azure Key Vault**. 2. Fill in the required fields: * **Name:** The display name for this provider. Must be unique. * **Vault URI:** The Vault URL from the Azure Portal Overview page. Read-only after the provider is created. * **Application ID:** The Client Id from the Azure Portal Overview page. * **Client Secret:** The client secret assigned to the application. * **Directory ID:** The Tenant Id from the Azure Portal Overview page. 3. Click **Add and Test** to save the provider and verify the connection. ![Add Azure Key Vault form](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-add-azure-key-vault.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=e6fa84dbc72abdeb21d11091e4bbcc44) Every secret in the connected Key Vault appears in Arc and is classified as **Encrypted**. You can add more than one Azure Key Vault provider if necessary. For example, you might need to connect to separate vaults for different environments. > [!NOTE] > > External providers (like Azure Key Vault) are only available on the Enterprise license tier. ### Adding an Environment Variable Provider The Environment Variable provider surfaces host machine environment variables as vault items. 1. In the **Vault Providers Config** pane, click **Add > Environment Variable**. The **Name** and **Type** are auto-generated. 2. Under **Included Patterns**, check the appropriate checkboxes and enter at least one pattern in **Standard Patterns**, **Encrypted Patterns**, or both. Each field accepts a comma-separated list of names or wildcard patterns using `*` and `?`. ![Add Environment Variable form](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-add-environment-variable.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=76a48218ba598ceba18c790348ce923e) 3. Click **Add** to save the provider. Only one environment variable provider is permitted. > [!NOTE] > > Only variables whose names match at least one pattern are visible in Arc, and at least one pattern (Standard or Encrypted) is required to add the provider. If the vault table appears empty after adding an Environment Variable provider, check that the patterns you entered actually match environment variable names that exist on the host. The pattern type controls how a matching variable is classified: * **Standard Patterns:** Matched variables are treated as Standard items and can be used in any non-sensitive field. * **Encrypted Patterns:** Matched variables are treated as Encrypted items and can only be used in sensitive fields. If a variable name matches both lists, an exact name match takes precedence over a wildcard match. If both matches are of the same kind (for example, two wildcards), Encrypted takes precedence over Standard. Variable names are matched case-insensitively on Windows and case-sensitively on other platforms. ## Managing Local Vault Items ### Adding Items to the Local Vault There are two ways to add an item to the local vault: 1. Navigate to **Settings > Vault** and click **Add Vault Item** at the top of the vault table. 2. Enter a value in a field elsewhere in Arc, click the key symbol, and select **Add to Vault**. ![Add vault item from a field using the key symbol](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-add-item-from-field.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=9d8e629fd7a15fff38d22e9fe5786b0f) Either option opens the **Add Vault Item** pane. Complete the following fields: * **Name:** The name of the vault item as it appears when you use it. * **Type:** The classification for the vault item. * *Standard* items can be used in all non-sensitive fields. Examples of standard items include URLs, usernames, and ports. * *Encrypted* items can only be used in sensitive fields. Examples of encrypted items include passwords and authtokens. * **Value:** The actual value that the vault item contains. For example, the URL, username, or password that you want to use when this vault item is selected. * **Labels:** (Optional) Labels for the vault item that make it easier to manage and search through large vaults. Separate each label with a comma or new-line character. ![Add vault item pane](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-add-modal.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=98a9bc9c55f8283481e449aa510939f3) Click **Add Vault Item** to save the item to your vault. ### Editing Vault Items To edit a vault item, click its name or hover over it, then click the ellipsis and choose **Edit**. The **Edit Vault Item** pane opens and allows you to edit the name, value, and labels for the item. Note that item Ids and types (standard or encrypted) cannot be edited after creation. To delete multiple vault items, check the box next to each item and click **Remove**. Items sourced from an external provider cannot be edited or removed in Arc. Hovering over an external item shows the tooltip: *"This item is managed by an external vault provider and cannot be edited or removed."* To change or remove an external item, update it in the originating system. ## Using Vault Items When an item is added to the vault, it can be used in fields that match the item type (standard or encrypted) and display a key symbol in the Arc UI: ![Key symbol on a connector field](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-key-symbol.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=de930b0e7894e164779e23a0088e8905) Click the key to open a list of matching vault values for the field. ![Vault item selection dropdown](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/vault-selecting-item.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=f3928b5f88a6270515bec79ae09b29f7) The list includes items from all configured providers. Each item shows the provider it comes from. Use the **Providers** filter at the top of the vault table to narrow the list to items from a specific provider. Select a value to populate it in the field. > [!NOTE] > > If you already have a value in the field, you can click **Add to Vault** to open the **Add Vault Item** pane as outlined [above](#adding-items-to-the-local-vault). ### Scripting Vault items can be used in ArcScript contexts. For more information, please see [Functions](../../../scripting/value-formatters/function-formatters#vault-itemname-ifnotexists-provider). > [!NOTE] > > Use caution when referring to encrypted vault items in scripting contexts to ensure that sensitive information is not included in the logs. # License Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/license-settings How to view and manage your CData Arc license, connector usage, and disk space usage. The **License** tab displays your licensing information, including which enterprise features you have enabled, the license expiration date, your product key, the version of CData Arc you are running, and your Node ID. ![License settings tab](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_license.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=31b9b85286185a7af99586b8bd82a91e) If you purchase a new license, you can install the license on this licensing tab using the **Add License** button. To extend a trial license, please contact [[arcsales@cdata.com](mailto:arcsales@cdata.com)](mailto:arcsales@cdata.com) with your Node ID. ## Connector Usage The **Connector Usage** section shows a breakdown of connector usage by category, providing a view of exactly how your current license is being utilized. ![Connector usage breakdown by category](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_license_connector_usage.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=5cba46123495a310d3b2f950e845c543) Click any category to see details on all instances of that connector type. This allows you to identify and manage connectors of any category, so you can free up allocations if necessary. ![Connector category detail view](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_license_connector_category.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=a23ddd1f116b176f895932efcb48eca7) ## Disk Space Usage The **Disk Space Usage** section shows exactly how much disk space you have free. If your disk is over 80% full, Arc displays a message to that effect, and notifies you that once you hit 90% full, automatic processing will be suspended. ![Disk space usage indicator](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_license_diskspace.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=dd76c6b017b46d3caa750d784c6f3992) # Migration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/migration How to import and export your full CData Arc application data and settings. The **Migration** portion of the **Advanced** page enables you to perform a full import, export, or migration of your application, including all of the global application data and settings. ![Migration section on the Advanced settings page](/26.4/images/settings_migration.png) Use the **Export** button to export global application data. Doing so exports your data in a `.arc` file. This file type stores global application data such as multiple workspaces and settings as well as users, roles, reports, and other global data. In a similar way, you can import application data by using the **Import** button to import a `.arc` file. You can also create and import `.arc` files from the workspaces grid on the **Flows** page. Use **Export Workspace** to export the workspaces you select, and use **Import Workspace** in the **Add Workspace** dropdown to import a `.arc` file. To export or import a single workspace as a `.arcflow` file, use **Export Flows** and **Import Flows** instead. For details about how to use the **Migration** settings to import or export your application, see [Importing and Exporting](../importing-exporting-flows). To migrate your application from the embedded database to an external database, use the [Application Database Migration](./app-db-migration) utility. # Network Access Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/network-access How to configure IP-based access restrictions and trusted/banned IP ranges in CData Arc. These network access settings provide additional security to CData Arc. If an IP address is banned, it cannot access the application at all (login access to the application, help, and Admin API pages is denied). Arc allows access to protected resources based on this priority list: 1. Global deny 2. User deny 3. User allow 4. Global allow 5. Default deny ![Network access IP range settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_network_access_ip.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=27dc0131b98e21e92961055cbb863a53) You can see in the list of **Protected Resources** that the **Admin API** and **Flow API** are *always* protected, and you cannot change that. Use the **Application UI** toggle to protect the UI. ![Network access global settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_network_access_global.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=1ac137846179de58b630de76c981c06c) Use the **Add Trusted IP Range** and **Add Banned IP Range** buttons to allow or ban specific IP ranges. You can provide single IP addresses, simple ranges (such as `203.0.113.*` or `203.0.113.1-255`), or Classless Inter-Domain Routing (CIDR) ranges. To edit an existing range, hover over it, click the ellipses, then click **Edit**. > [!NOTE] > > Because you can lock yourself out of the application with this setting, it detects when this might happen and does not allow you to save changes. # Performance Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/performance-settings How to configure global Automation service settings for parallel processing and worker pools in CData Arc, and how to analyze worker pool performance. This section allows you to define global settings for the [Automation](../../architecture#automation) service. ![Performance settings page](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_performance.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=8fd458b22db9568f8df4ca7f48b3b74f) | Setting | Description | | - | - | | **Parallel Processing** | When parallel processing is enabled, Arc allocates a pool of workers to process messages for multiple connectors in parallel, as opposed to performing processing for each connector in series. Parallel processing improves processing throughput across multiple transfers and message processing across multiple stages. The following options only appear when parallel processing is enabled. | | **Worker Pool** | Establishes the global maximum number of threads that the application can assign to all connectors (combined) at the same time. When a thread finishes its assignment in a specific connector, it is recycled back into the pool. The hardware resources available on the host machine determine the upper limit for this setting. A larger worker pool consumes more system resources while Arc is processing messages. To diagnose how your worker pool is performing, see [Analyze Worker Pool](#analyze-worker-pool). | | **Max Workers per Connector** | The default number of workers that a connector uses to process transmissions. This setting can be overridden on an individual basis on the **Advanced** tab of each connector. Connectors that use session communication such as FTP, SFTP, and OFTP default to 1 worker, since multiple files might be transferred over the same connection. | | **Max Files per Connector** | The default number of files that workers assigned to a connector process before the workers are reallocated to other connectors. This can be overridden on an individual basis on the **Advanced** tab of each connector. When a large number of connectors are configured, lower values here ensure that processing is distributed evenly across all connectors. | See [Maximizing Performance](../../architecture#maximizing-performance) for more information. ## Analyze Worker Pool The **Analyze Worker Pool** button collects diagnostic logs from the worker pool for a set period of time and runs an AI analysis on them. The analysis reports on the health of the worker pool, identifies performance findings, and recommends configuration changes. Arc never applies recommendations automatically. Review each recommendation and make the changes yourself. This feature requires an AI provider. If an AI provider is not configured in [AI Settings](./ai-settings), the button is disabled. Hover over the info icon next to the button to see why. Only members of the [Admin role](./user-roles#admin-role) can start, cancel, and remove analyses. Members of the Standard and Support roles can open the pane and view previously run analyses, but cannot start, cancel, or remove them. ### Starting an Analysis 1. Click **Analyze Worker Pool** to open the **Analyze Worker Pool Sessions** pane. The pane lists any previous analysis sessions. ![Analyze Worker Pool Sessions pane](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_performance_analyze_worker_pool.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=acc4c55d0ee2f40525bede71154e09d0) 2. Under **New Analysis**, click **Start Analysis** to open the **Configure Analysis** pane. ![Configure Analysis pane](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_performance_configure_analysis.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=920a0a6cf7053b34ff52492ae0f00a3e) 3. Set the following options: | Option | Description | | - | - | | **Log Collection Duration** | How long Arc collects diagnostic logs before running the analysis. Choose 1, 2, 3, 5, or 10 minutes. The default is 5 minutes. Longer windows capture more activity but delay the results. | | **Log Analysis Verbosity Levels** | Controls how much detail appears in the results. The default is **Summary**. **Summary**: Health assessment and key limitations only. **Standard**: Adds status information and identified performance findings. **Detailed**: Full analysis including raw metrics and optimization recommendations. | > [!NOTE] > > The verbosity level is set when the analysis starts and cannot be changed afterward. To see more detail, run a new analysis with a higher verbosity level. 4. Click **Start Analysis**. The analysis runs in two phases: **Collecting logs**, which shows the time remaining, followed by **Analyzing logs**. You can close the pane while the analysis runs. When it finishes, Arc displays an in-app notification in the page header. Click **Go to Results** to open the analysis, or **Go to Details** if the analysis ended in an error. To stop an in-progress analysis, click **Cancel Analysis** and confirm. ### Viewing Results Each completed analysis is saved and listed under **Previous Analysis Sessions** in the **Analyze Worker Pool Sessions** pane, along with its file count and size. Click a session to open its **Worker Pool Log Analysis** report. Click the copy icon in the report header to copy the full analysis. ![Worker Pool Log Analysis results](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_performance_worker_pool_analysis_results.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=899cb8ef182ed654dc0706298ba809ef) The sections that appear in the report depend on the verbosity level you selected: | Section | Description | | - | - | | **Summary** | The overall health of the worker pool over the collection window. | | **Status** | What the analyzer read to produce this report. | | **Findings** | Behavior the analyzer identified in the collected logs. | | **Limitations** | What the collected logs were not able to answer. | | **Metrics** | Values the analyzer computed from the collected logs. | | **Recommendations** | Configuration changes the analyzer suggests. Each recommendation lists the **Setting**, its **Config Level**, the current and recommended values, the **Rationale**, and the **Expected Impact**. | | **Additional Details** | Anything else the analyzer reported that does not fit the sections above. | To delete a saved analysis, hover over the analysis and click the trashcan icon. Confirm your selection and click **Remove**. Only members of the Admin role can remove analyses. # Proxy Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings How to configure global inbound and outbound proxy settings in CData Arc. The **Proxy Settings** portion of the **Security** page contains global inbound and outbound proxy settings. Connectors that support an outbound proxy (such as [AS3](../../../connectors/as3)) default to these **Outgoing** settings, but you can configure individual connectors with separate proxy settings on the **Advanced** tab of some other connectors (for example, [FTP](../../../connectors/ftp) and [SFTP](../../../connectors/sftp)). ![Proxy settings page](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_proxy.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=7c4bb54cb957829351e8bb2204f0293a) ## Incoming Proxy Settings | Setting | Description | | - | - | | **HTTP Forwarded Header** | Use this when Arc is behind a proxy or routing agent to record the IP address of original incoming requests in the access log. When specified, the application first checks for a header with this value to record as the remote IP column's value in the log. | ## Outgoing Proxy Settings | Setting | Description | | - | - | | **Proxy Type** | The protocol used by a proxy-based firewall. | | **Proxy Host** | The name or IP address of a proxy-based firewall. | | **Proxy Port** | The TCP port for a proxy-based firewall. | | **Proxy User** | The user name to use to authenticate with a proxy-based firewall. | | **Proxy Password** | A password used to authenticate to a proxy-based firewall. | | **Authentication Scheme** | Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**. | # Security Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/security Overview of security configuration options in CData Arc, including SSO, sign-in settings, network access, Admin API, and proxy settings. The **Security** tab holds several categories of security-related configuration options, each of which has its own documentation topic: * [Admin API Options](./admin-api-settings) covers Cross-Origin Resource Sharing (CORS) settings and authentication token configuration for browser-based clients and API access. * [Encryption at Rest](./encryption-at-rest) supports application-level encryption at rest for [message data](../activity#messages-tab). It addresses data-protection requirements for standards such as PCI DSS, HIPAA, GDPR, and SOX. * [Network Access](./network-access) options control IP-based access restrictions and provide additional security layers by managing trusted and banned IP ranges for protected resources. * [Proxy Settings](./proxy-settings) options enable you to configure global inbound and outbound proxy settings for network traffic routing and security. * [Sign In Settings](./sign-in-settings) options allow you to define global authentication requirements, password policies, and user account security measures including two-factor authentication and inactivity timeouts. * [Single Sign-On (SSO) and User Provisioning](./single-sign-on) covers options for configuring single sign-on via SAML 2.0 or OpenID Connect authentication protocols, including integration with identity providers such as Microsoft Entra ID and Okta. # Settings Page Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/settings-page Overview of the global settings and user management tools available in CData Arc. The **Settings** page contains global settings and user management tools. To launch the page, click the gear icon on the navbar. ## User Management and Roles The **Users** tab contains a table of the users that are authorized to access the application. This table displays the credentials, roles, and status for each user. Use this tab to add users, change a user's status, or manage their permissions and auth tokens. The **Roles** tab displays a table of all roles in the application, including the three built-in roles and all custom roles. From here you can define additional roles and the policies that make up those roles, and assign users to roles. A policy is a set of permissions that determine exactly how users can interact with the application. For more information, see [Users](./users) and [User Roles](./user-roles). ## Certificates The [Certificates](./certificates) tab displays a sortable table of all certificates that are used and managed by the application. Use this tab as a reference to quickly determine which certificates are in use by which connectors in the flow. ## Alerts Use the **Alerts** tab to configure intelligent monitoring that keeps you informed with timely alerts for system health, certificate expirations, processing errors, and critical events before they impact your operations. You can also use it to configure [custom alerts](./alerts#custom-alerts). ## Connections The [Connections](./connections) tab lists configured connection parameters that can be shared among multiple CData Arc connectors. ## Security The [Security](./security) tab contains several sets of security related-settings. * [Single Sign-On](./single-sign-on) (SSO) enables you to set up SSO via either the SAML 2.0 or OpenID authentication protocols. * [Sign In Settings](./sign-in-settings) enable you to configure global sign-on requirements that govern authentication and password management. * [Network Access](./network-access) settings provide additional security to Arc by allowing you to set or ban specific IP addresses or ranges. * [Admin API](./admin-api-settings) includes settings related to Cross-Origin Resource Sharing (CORS) and whether auth tokens can be passed in query string parameters for authentication. * [Proxy Settings](./proxy-settings) enable you to configure global inbound and outbound proxy settings. ## Vault The [Vault](./global-settings-vault) tab enables you to add and manage values that you can re-use throughout the Arc admin console. ## Advanced The [Advanced](./advanced-settings) tab holds several categories of configuration options: * [Cleanup Settings](./cleanup-settings) cover options related to database and log file cleanup intervals. These options are designed to improve application optimization and free up disk space. * [Performance Settings](./performance-settings) options allow you to define global settings for the [Automation](../../architecture#automation) service. * [AI Settings](./ai-settings) options control the behavior of AI features. * [Migration](./migration) options enable you to perform a full import or export of your application, including all of the global application data and settings. * [Git Version Control](./git-version-control-settings) enables you to set up a Git repository to aid in configuration management. * [Additional Settings](./additional-settings) includes a number of different settings, including whether to show beta and deprecated connectors, the default log level, time format settings, and more. ## License The [License](./license-settings) tab displays your licensing information. Use it to quickly review your connector usage, see which enterprises features you have enabled, check your disk space usage, and more. # Sign In Settings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/sign-in-settings How to configure global authentication requirements, password policies, and account security measures in CData Arc. To help safeguard user accounts and ensure compliance with organizational security policies, the application provides configurable global sign-on requirements that govern authentication and password management. These settings control factors such as two-factor authentication, inactivity timeouts, and detailed password complexity rules, giving administrators the flexibility to balance ease of access with strong security. | Setting | Description | | - | - | | **Two-Factor Authentication (2FA)** | Specifies whether users can or must enable 2FA for their account. See [Two-Factor Authentication](../2fa) for details. | | **Automatically Disable Users after Inactivity** | Automatically disables user accounts that have been inactive for the period specified in **Inactive Days Until Disabling Account**. Disabled users cannot log in until they are manually re-enabled by an administrator. | | **Inactive Days Until Disabling Account** | The number of days of inactivity before an account is disabled. A value of `-1` means the account is never disabled. | | **Password Expiration** | Whether users' account passwords must be reset at certain intervals. | | **Days Before Password Expires (30 days minimum)** | The frequency at which users must reset their passwords. Only appears when **Password Expiration** is enabled. | | **Expiring Password Alert** | The number of days before password expiration that the user is reminded to change their password. Only appears when **Password Expiration** is enabled. | | **Reuse Password Restriction** | The number of cycles of password resets before a user can reuse a previous password. | | **Minimum Password Length** | The minimum number of characters required to set a password. | | **Minimum Password Special Characters** | The minimum number of special characters (such as `?`, `&`, or `#`) required. | | **Minimum Password Lowercase Characters** | The minimum number of lowercase characters. | | **Minimum Password Uppercase Characters** | The minimum number of uppercase characters. | | **Minimum Password Numeric Characters** | The minimum number of numeric characters. | # SSO and User Provisioning Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/single-sign-on How to configure single sign-on via SAML 2.0 or OpenID Connect in CData Arc, including Microsoft Entra ID and Okta integration. CData Arc supports single sign-on (SSO) via either the SAML 2.0 or OpenID authentication protocols. This feature enables identity providers such as Microsoft Entra ID (formerly Azure Active Directory) or Okta, which support these protocols, to serve as SSO platforms for Arc. ## Overview Enabling SSO requires that you configure a group of settings on the **Security** tab of the **Settings** page. When you configure the SSO settings, users that are created within Arc should be given a federation Identifier (Id). This federation Id associates a Arc username with an identity provider entity. When the Arc user attempts to log in to the application, it is redirected to the SSO platform where the credentials for the appropriate identity must be entered. If Just-in-Time (JIT) user provisioning is enabled (also from the **Security** tab), a new user account is automatically created in Arc at the time of login, based on attributes that are received from the identity provider (for example, the user's name and email address). This identity and access management (IAM) process streamlines onboarding by eliminating the need for manual account creation. For more details, see [Just-in-Time (JIT) User Provisioning](#just-in-time-jit-user-provisioning). After SSO is configured and Arc users have a federation Id value to associate them with an identity provider, any additional advanced configuration for SSO user management should be performed directly in the SSO platform, as explained in the following sections. > [!NOTE] > > * Once SSO is enabled, existing users who are not members of the [Admin role](./user-roles#admin-role) cannot log in unless they have a federation Id (described below) on their account. Admin users can still log in without SSO in case the SSO settings need to be reconfigured. > > * Currently, Arc supports only individual users, not groups of users. If an SSO platform provides access for a group of users, each individual user within that group must be added as a user on the Arc **Settings** page in order to log in. Each user should reference the federation Id from the identity provider. The following sections explain how to set up SSO configuration through two identity providers, Microsoft Entra ID and Okta, with the OpenID Connect and SAML 2.0 authentication protocols. ## SSO Settings This section describes each of the SSO settings and what they represent in Arc. The details of these settings might differ based on your IdP. ### OpenID Connect Settings | Setting | Description | | - | - | | **Callback URL** | The callback URL is configured in the IdP (not Arc) to indicate where users should be returned once they perform the authentication against the IdP. | | **OAuth Client Id** | The client Id associated with the OAuth application created in the IdP. | | **OAuth Client Secret** | The client secret associated with the OAuth application created in the IdP. | | **Audience URIs** | The intended recipient of SSO login tokens. This ensures that tokens generated by the SSO platform are intended for use by Arc. Set this value to the URI that identifies Arc as a trusted application in your SSO platform (for example, the *Application Id* that the SSO platform generated for Arc). To accommodate multiple URIs, supply a comma-separated list. | | **Discovery URL** | Required to import settings from your SSO provider. | | **Issuer Certificate** | The public portion of the certificate that the IdP uses to sign JWT tokens. This helps ensure that only tokens generated by the expected IdP are accepted. | | **Token Issuer Identifier** | The identifier used by the IdP when generating JWT tokens. Arc validates tokens to ensure that they include this identifier. This helps ensure that only tokens generated by the expected IdP are accepted. | | **Authorization URL** | The URL where the user should perform authorization via the IdP. When an SSO user attempts to login in to Arc, the application redirects the user to this URL where they enter their SSO platform credentials. | | **Token URL** | The URL where a secure authentication token can be obtained from the IdP. | | **Logoff URL** | Specifies where the Arc user is redirected after logging out of the application. | | **Default Scopes** | A space-separated list of scopes (sets of permissions) to request from the IdP. Typically the scope(s) should at least include `profile openid` to request identity verification. | | **Key Claim** | The value from the IdP to treat as the federation Id. In other words, this specifies the claim to use as the federation Id when mapping users from the IdP to Arc usernames. | | **Token Signature Algorithm** | Which signature algorithms are accepted from signed JWT tokens. Unsigned tokens are never accepted. | | **Just-in-Time (JIT) Provisioning** | Whether a new user account is created automatically in Arc when a user first logs in. User details are based on the attributes provided in the claim. | | **Group Mapping** | Maps SSO group identifiers to Arc [roles](./user-roles). When a user logs in, Arc matches the groups in the claim against these mappings and assigns the roles from every matching group. The **Default** row sets one or more roles (Support, Standard, Admin, or Custom) to assign when no mapping matches. If you select **No Default Role**, users whose claims do not map to a role are denied sign-in. No user is created, and any existing user's roles are removed. For more details, see [JIT Configuration in Arc](#jit-configuration-in-arc). | ### SAML 2.0 Settings | Setting | Description | | - | - | | **Assertion Consumer Services (ACS) URL** | The endpoint on the Arc application where the IdP sends the SAML authentication response after a user successfully logs in. This is similar to the Callback URL setting in OpenID. | | **Audience URI** | A unique identifier (typically the application's Entity ID) that the SAML assertion is intended for. Used to validate that the response is meant for your application. | | **SAML Discovery URL** | The endpoint used to determine which IdP should be used for authentication. Required to import settings from your SSO provider. | | **Issuer Certificate** | The public portion of the certificate that the IdP uses to sign JWT tokens. Helps ensure that only tokens generated by the expected IdP are accepted. | | **Entity ID** | A globally unique identifier used to identify an IdP or service provider in a SAML authentication process (often formatted as a URI). | | **SSO URL** | The IdP endpoint where the service provider redirects the user to initiate the SAML authentication flow. | | **Logoff URL** | Specifies where the Arc user is redirected after logging out of the application. | | **Just-in-Time (JIT) Provisioning** | Whether a new user account is created automatically in Arc when a user first logs in. User details are based on the attributes provided in the SAML assertion. | | **Group Mapping** | Maps SSO group identifiers to Arc [roles](./user-roles). When a user logs in, Arc matches the values of the **Role** attribute in the SAML assertion against these mappings and assigns the roles from every matching group. The **Default** row sets one or more roles (Support, Standard, Admin, or Custom) to assign when no mapping matches. If you select **No Default Role**, users whose claims do not map to a role are denied sign-in. No user is created, and any existing user's roles are removed. For more details, see [JIT Configuration in Arc](#jit-configuration-in-arc). | ## Configuring Microsoft Entra ID for SSO The following sections explain how to set up Microsoft Entra ID for SSO via either OpenID Connect or SAML 2.0. ### OpenID Connect Configuration Single sign-on (SSO) with Microsoft Entra ID streamlines access to CData Arc, enabling just-in-time provisioning at the time of login. The following steps explain how to set up Microsoft Entra ID for SSO with the OpenID Connect protocol. 1. Log in to the Azure portal and open Microsoft Entra ID. 2. Log in to CData Arc and select **Settings > SSO** to open the **Single Sign-On (SSO) Settings** dialog box. 3. Browse to **Entra ID > App registrations > New registration** and create an application registration for Arc. Follow the prompts to complete the process. 4. In your app registration, set the redirect URL to the **Callback URL** value that is specified in the **Single Sign-On (SSO) Settings** dialog box in Arc, as shown below. ![Redirect URL configuration in Azure app registration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_openid-connect-configuration_1.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=d02e2f8566afa89bdf052391bbe4ed50) 5. In the **Single Sign-On (SSO) Settings** dialog box in Arc, specify the following properties: 1. Set the **Client ID** property to the value of **Application (client) Id** that is found in the new app registration shown below. ![Application client ID in Azure app registration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_openid-connect-configuration_2.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=23c1675818d56b58620f266fee6c8694) 2. Set the **Client Secret** property to the value of the new client secret that you created in the **Client credentials** section of your app registration. 3. Set **Discovery URL** to the **OpenID Connect MetaData document** value from the **Endpoint** page in your app registration. Then, click **Import** to import the remaining settings into Arc. 6. Click **Save and Test** in Arc to verify the SSO configuration. Arc opens a new tab and prompts you to sign in to your Microsoft account. If the test succeeds, Arc displays a success message along with claim details. 7. (Optional) Update your Arc user account with your federation Id. 1. Copy the federation Id from the **Single Sign-On (SSO) Settings** dialog box in step 5. 2. Navigate to **Settings > Users**. Then, locate your user account and click **Edit**. 3. Add the federation Id to your user account. Then click **Save** to save your change and exit the dialog box. At this point, you have completed the basic OpenID Connect SSO setup, allowing your users to log in to Arc through Microsoft Entra ID. ### SAML 2.0 Configuration With SAML-based single sign-on (SSO), users can seamlessly authenticate through their organization's identity provider, enabling just-in-time provisioning at the time of login. The following steps explain how to set up Microsoft Entra ID SSO with the SAML 2.0 protocol. 1. Log in to the Azure portal and open Microsoft Entra ID and navigate to the **Enterprise applications** page. 2. Select **New application > Create your own application**. In the **Create your own application** dialog box: 1. Enter a name for your application (for example, **CData Arc**). 2. Specify what you want to do with your application by selecting one of the options under **What are you looking to do with your application?** 3. Click **Create**. 3. In your newly created application, select **Set up single sign on**. 4. Select **SAML** as the sign-on method. 5. Log into CData Arc and select **Settings > SSO**. On that **SSO** tab, click **Configure** in the **Single Sign On (SSO) Settings** section. This action displays the **Single Sign On (SSO) Settings** dialog box. 6. Select **SAML 2.0**. This selection displays the Assertion Consumer Services (ACS) URL and the audience URI. ![SAML 2.0 SSO settings dialog showing ACS URL and audience URI](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_saml_1.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=e293727b8cd8bc2568702d83c71a9f2a) > [!NOTE] > > Keep this dialog box open because you need to return to it later in these steps. 7. In Entra ID, set **Reply URL (Assertion Consumer Service URL)** to the **Assertion Consumer Services (ACS) URL** value in Arc. Then, set **Identifier (Entity ID)** in Entra ID to the **Audience URI** value in Arc. > [!NOTE] > > Leave the **Sign on URL** text box empty if you plan to use sign-on initiated by your identity provider. 8. In the Arc **Single Sign-On (SSO) Settings** dialog box, set **Discovery URL** to the **App Federation Metadata URL** value that is found in Entra ID. Then, click **Import** to import the remaining settings into Arc. 9. Click **Save and Test** in Arc to verify the SSO configuration. Arc opens a new tab and prompts you to sign in to your Microsoft account. If the test succeeds, Arc displays a success message along with claim details. > [!NOTE] > > If you use the [.NET edition](../../installation/windows-edition), see [Known SAML 2.0 Configuration Issue for .NET](#known-saml-2-0-configuration-issue-in-net) for information on a .NET issue that can affect the test. 10. (Optional) Update your Arc user account with your federation Id. 1. Copy the federation Id from the **Single Sign-On (SSO) Settings** dialog box in step 9. 2. Navigate to **Settings > Users**. Then, locate your user account and click **Edit**. 3. Add the federation Id to your user account. Then click **Save** to save your change and exit the dialog box. At this point, you have completed the basic SAML SSO setup, allowing your users to log in to Arc through Microsoft Entra ID. ## Configuring Okta for SSO The following sections explain how to set up Okta for SSO via either OpenID Connect or SAML 2.0. ### OpenID Connect Configuration Single sign-on (SSO) with Okta streamlines access to CData Arc, enabling just-in-time provisioning at the time of login. The following steps explain how to set up Okta for SSO with the OpenID Connect protocol. 1. Log in to the Okta Admin Console and select **Applications > Applications > Create App Integration**. ![Okta create app integration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_openid-configure-okta_1.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=00d225b4fa05f6e164ea8320803d9bb1) 2. In the **Create a new app integration** dialog box, select **OIDC - OpenID Connect** for **Sign-in method** and **Web Application** for **Application type**. Then click **Next**. ![Create new app integration dialog in Okta](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_openid-configure-okta_2.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=d4534a23e7bdacfd39cccf0931a36daa) 3. Log in to CData Arc and select **Settings > SSO** to open the **Single Sign-On (SSO) Settings** dialog box. 4. In Okta, set the redirect URI to the **Callback URL** value that is specified in the **Single Sign-On (SSO) Settings** dialog box in Arc, as shown below. ![Redirect URI configuration in Okta](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_openid-configure-okta_3.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=34c3df9d2ebf4278e292d8f2079806d3) Then, click **Next** to create your application. 5. Locate and copy the Issuer URL, as follows. 1. Click the **Sign-On** tab for your application in the Okta Admin Console. 2. Scroll to the **OpenID Connect ID Token** section. 3. Copy the URL from the **Issuer** field. > [!NOTE] > > If you want to sign in through an authorization server, you must obtain the Issuer URL for that particular server. In the Okta Admin Console, click **Security** in the left navigation pane, and then select **API** to display the **Authorization Servers** list. Locate your authorization server and copy the URL from the **Issuer** field. 6. In the **Single Sign-On (SSO) Settings** dialog box in Arc, specify the following properties: 1. Set the **Client ID** and **Client Secret** properties to the corresponding values that are found on the **General** tab in Okta. 2. Set **Discovery URL** to the Issuer URL that you copied previously, and append the OpenID Connect Discovery endpoint (`.well-known/openid-configuration`). **Example:** `https://MyOrganization.okta.com/.well-known/openid-configuration` Then, click **Import** to import the remaining settings into Arc. 7. Click **Save and Test** in Arc to verify the SSO configuration. Arc opens a new tab and prompts you to sign in to your Okta account. If the test succeeds, Arc displays a success message along with claim details. 8. (Optional) Update your Arc user account with your federation Id. 1. Copy the federation Id from the **Single Sign-On (SSO) Settings** dialog box in step 6. 2. Navigate to **Settings > Users**. Then, locate your user account and click **Edit**. 3. Add the federation Id to your user account. Then click **Save** to save your change and exit the dialog box. At this point, you have completed the basic OpenID Connect SSO setup, allowing your users to log in to Arc through Okta. ### SAML 2.0 Configuration With SAML-based single sign-on (SSO), users can seamlessly authenticate through their organization's identity provider, enabling just-in-time provisioning at the time of login. The following steps explain how to set up Okta for SSO with the SAML 2.0 protocol. 1. Log in to the Okta Admin Console and select **Applications > Create App Integration**. This step opens the **Create a new app integration** dialog box. 2. Select **SAML 2.0** as the sign-in method. Then, click **Next**, which opens the **Create SAML Integration** dialog box. 3. Enter a descriptive name (for example, **CData Arc**) for your application in the **App Name** text box. You can also add a logo for your application, if you choose. Then, click **Next**. 4. In the Arc application, click the **SSO** tab on the **Settings** page. On that tab, click **Configure** in the **Single Sign On (SSO) Settings** section. This action displays the **Single Sign On (SSO) Settings** dialog box. 5. Select **SAML 2.0**. This selection displays the Assertion Consumer Services (ACS) URL and the audience URI. ![SAML 2.0 SSO settings dialog showing ACS URL and audience URI](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/settings_sso_saml_1.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=e293727b8cd8bc2568702d83c71a9f2a) > [!NOTE] > > Keep this dialog box open because you need to return to it later in these steps. 6. In Okta, enter the **Assertion Consumer Services (ACS) URL** value from Arc into the **Single sign on URL** text box. Then, enter the **Audience URI** value from Arc into the **Audience URI (SP Entity ID)** text box. You can leave the default settings for the remaining Okta fields. 7. In Arc, set **Discovery URL** to the domain of your registered Okta organization, followed by `/oauth2/default/.well-known/openid-configuration`. **Example:** `https://trial-1234567.okta.com/oauth2/default/.well-known/openid-configuration` Then, click **Import** to import the remaining settings into Arc. 8. Click **Save and Test** in Arc to verify the SSO configuration. Arc opens a new tab and prompts you to sign in to your Okta account. If the test succeeds, Arc displays a success message along with claim details. > [!NOTE] > > If you use the [.NET edition](../../installation/windows-edition), see [Known SAML 2.0 Configuration Issue for .NET](#known-saml-2-0-configuration-issue-in-net) for information on a .NET issue that can affect the test. 9. (Optional) Update your Arc user account with your federation Id. 1. Copy the federation Id from the **Single Sign-On (SSO) Settings** dialog box in step 8. 2. Navigate to **Settings > Users**. Then, locate your user account and click **Edit**. 3. Add the federation Id to your user account. Then click **Save** to save your change and exit the dialog box. At this point, you have completed the basic SAML SSO setup, which enables you to log in to Arc through Okta. ## Just-in-Time (JIT) User Provisioning As mentioned [earlier](#overview), *Just-in-Time user provisioning* enables Arc to create and manage user accounts automatically at first login by using information that is provided by your identity provider. JIT provisioning is particularly useful when users access Arc through an identity provider for the first time, and the application receives a secure message confirming their identity. The resulting account is assigned the roles mapped to the user's groups in the claim or, if no mapping matches, the default role specified in the SSO settings. This functionality streamlines user onboarding and ensures account details remain consistent with identity claims. When a user logs in to Arc by using SAML or OpenID Connect, Arc searches for that user via a federation Id. * If a user **does not** exist, Arc creates a new user automatically by using the information that is provided in the authentication claim, provided that a role can be resolved (see [JIT Configuration in Arc](#jit-configuration-in-arc)). * If a user **does** exist, Arc updates the user's username and email if they do not match the claim values. If no role can be resolved and you have selected **No Default Role** in **Group Mapping**, Arc removes the user's roles and denies login. ### Requirements and Mappings in your Identity Provider Identity-provider requirements vary depending on whether you use SAML 2.0 or OpenID Connect. This section explains claim requirements for each authentication method. **OpenID Connect** * By default, subcontrols are mapped to the user's role and the claim is mapped to the federation Id. As an option, **admin** users can use a different field (for example, **oid**) by setting the **Key Claim** property in Arc. * The **Email** claim maps to **Email Address** in Arc. * The **preferred\_username** claim maps to **name** in Arc. **SAML 2.0** * The **NameID** and **Email** claims are required. * (Optional) You can add the **Name** claim, which maps to **Username** in Arc. * (Optional) You can add the **Role** claim. Arc matches its values against **Group Mapping** to assign the user's roles. ### JIT Configuration in Arc To enable and configure JIT provisioning in Arc: 1. Enable JIT provisioning, as follows: 1. Select **Settings > SSO** in Arc. 2. Click the tablet and pencil ![tablet and pencil](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_expressioneditor_icon.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=ca9752121351859d881672fcb4d70060) icon to open the **User Provisioning** dialog box. 3. Select **Enable** under **JIT Provisioning**. 4. In the **Group Mapping** section, select one or more [roles](./user-roles) from the **Arc Roles** list for the **Default** row. If you select **No Default Role**, Arc denies login to users whose claims do not map to a role. 5. (Optional) To assign roles based on a user's SSO group, click **Add Mapping**, enter the SSO group identifier, and select the roles to assign to members of that group. 6. Click **Save** to save your selection and exit the dialog box. When a user logs in, Arc resolves the user's roles in the following order: 1. Arc matches the group values in the claim (the `groups` claim for OpenID Connect, or the **Role** attribute for SAML) against **Group Mapping**. The user gets the roles from every matching group. 2. If no mapping matches, Arc assigns the roles in the **Default** row. 3. If the **Default** row is **No Default Role**, login is denied. Arc does not create a user account, and any existing user's roles are removed. Arc reapplies the resolved roles each time the user logs in, whether they come from a mapping or the **Default** row, so an admin user cannot override them in Arc. > [!WARNING] > > If Arc cannot resolve a role for a user and you selected **No Default Role**, the user sees the error *Access denied. No role is available for this account. Contact your administrator.* Manually assigning a role to the user in Arc does not resolve this, because the next SSO login removes the role again. To resolve the issue, make sure your identity provider sends the user's group in the claim and add a group mapping for that group, or select a default role in the **Default** row of **Group Mapping**. ## Known SAML 2.0 Configuration Issue in .NET When you configure SAML 2.0 in the .NET edition, clicking the **Save and Test** button appears to authenticate successfully but then displays an error. This is a known .NET platform issue, and your actual SAML login is unaffected. These settings are not included in Web.Config by default because relaxing the SameSite cookie restrictions can increase exposure to cross-site request forgery (CSRF) attacks. Enabling this workaround trades stricter default security for compatibility with certain SAML 2.0 login flows in .NET. Follow these steps to work around the issue: 1. Add the following to your Web.Config file (by default, the file is in C:\Program Files\CData\CData Arc\www): * `cookieSameSite="None" requireSSL="true"` in `authentication mode="forms"` * `sameSite="None" requireSSL="true"` in `httpCookies` * `cookieSameSite="None" cookieless="UseCookies"` in `sessionState` 2. Restart Arc with HTTPS. This sample Web.Config file shows the necessary updates: ```xml theme={null} ``` # User Roles Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/user-roles How to create and manage roles, policies, and permissions in CData Arc. The **Roles** tab displays a table of all roles defined in the application, along with the associated workspaces and a link to all users who can assume each role. Roles enable you to restrict users to only performing certain actions in certain workspaces. The roles assigned to a user affect everything that they see in Arc, including but not limited to: * [Workspaces](../../../flows/workspaces) * [Reports](../reports) * [Dashboard data](../dashboard) * [Logs](../activity) To create and manage roles, navigate to the [Settings page](./settings-page), then the **Roles** tab. For information about creating and managing the users who are assigned these roles, see [Users](./users). Every installation of Arc includes three built-in roles: * [Admin](#admin-role) * [Standard](#standard-role) * [Support](#support-role) These are global roles that cannot be edited or deleted. They apply to all workspaces. However, you can define [custom roles](#custom-roles) which limit each user's permissions. Custom roles are made up of *policies*, which provide the most granular control over exactly what users can see or do in Arc. Custom roles also identify which workspaces, Flow APIs, connections, and certificates users can see and interact with. The following sections describe each built-in role, explain how to define new policies and roles, and provide an example policy. Use the [comparison table](#user-roles-comparison-table) for help building your own policies and roles. ## Admin Role The **Admin** role provides full control over the application. An admin can create new [flows](../../../flows), delete existing flows, change [Profile](../profiles) settings, and perform every other operation supported by the console. In addition, only admins can view the [Audit Log](../activity#audit-logs), which records all changes made by any user in the application in a [tamper-evident manner](../activity#tamper-evident-logging). ## Standard Role Members of the **Standard** role can create, edit, and delete connectors and flows. However, they cannot change any application-wide settings, such as those exposed in the [Profile](../profiles) page. Standard users can upload new files into flows, and can upload public certificates for use by connectors. They cannot upload private certificates that are set in a Profile. ## Support Role The **Support** role is a read-only role; members of this role cannot create or delete flows, or change application settings. **Support** users *can* send files through existing flows using the *Send* operation, but cannot upload new files (in other words, they can only process files already present in the **Transactions** tab of a connector). ## Custom Roles Custom roles give you more granular control over what each user can see or do in Arc. Custom roles are defined by creating policies, which specify the exact permissions members of that role have. A single policy can grant permissions across three areas: workspaces (including Flow APIs and connectors), connections, and certificates. They also identify which specific workspaces, connections, and certificates users can see and interact with. ### Creating Roles and Policies 1. To create a custom role, navigate to the [Settings page](./settings-page), then the **Roles** tab. 2. Click **Add Role**. Give the role a meaningful name and description, then click **Add Role**. 3. To define the policy, click **Add Policy**. 4. A policy is organized into permission sections—**Workspace Permissions** (which also includes Connectors and Flow API access), **Connection Permissions**, and **Certificate Permissions**. Use the toggle next to each section to enable it, then set the allowed actions and access scope for that section as described below. For each type of access, use the corresponding **All current and future** toggle to grant access to every current and future item of that type, or clear the toggle and select a specific subset. 5. To add another policy, click **Add Policy** and repeat the previous steps. 6. When you are finished, click **Save**. The following image shows a role with two policies. ![Role with two completed policies](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/role_policy_example_263.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=fa57364ae004c9e8ac5c9c7cb1b6c67b) ### Workspace Permissions #### Workspace Configuration Check the checkboxes next to the **Allowed Workspace Actions** you want to assign, then use the **Workspace Access** section to assign the policy to one or more workspaces. To assign it to all workspaces, use the **All current and future workspaces** toggle. #### Connector Configuration The **Connector Configuration** area within this section controls what users can do with the connectors in the selected workspaces. Check the checkboxes next to the **Allowed Connector Actions** you want to assign, then use **Connector Access** to scope the policy to specific connector types, or to all current and future connectors. #### Flow API Configuration Within the **Workspace Permissions** section, the **Flow API Configuration** area controls what users can do with [Flow API](../../../flows/flow-api) endpoints. Check the checkboxes next to the **Allowed Flow API Actions** you want to assign. Use **Flow API Access** to scope the policy to specific Flow APIs, or use the **All current and future Flow APIs** toggle to grant access to every Flow API within the selected workspaces. > [!NOTE] > > The **Invoke** action is independent of workspace access. A [service account](./users#user-types) that is granted only the **Invoke** permission can call a Flow API endpoint without having any other access to—or visibility into—the Arc UI. This lets you expose a Flow API endpoint to an automated client without giving it broader access to the application. The following image shows **Policy 1** with completed **Workspace**, **Connector**, and **Flow API** permissions. ![Policy 1 with workspace, connector, and flow API permissions completed](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/roles_policy1_complete_263.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=df76d641c3b208f92173c6dd4e879e4c) ### Connection Permissions The **Connection Permissions** section controls the connection lifecycle: creating, updating, and deleting connections. Check the checkboxes next to the **Allowed Connection Actions** you want to assign. The ability to view a connection is implicit whenever the policy grants any connection permission. Use **Connection Access** to scope the policy by connection name, by connection type (such as SFTP, Amazon S3, or Salesforce), or both. When you set both a name filter and a type filter, they are combined—only connections that match *both* are in scope. To grant access to every connection regardless of these filters, use the **All current and future connections** toggle, which overrides the name and type filters. The following image shows a policy with completed **Connection** permissions. ![Policy with connection permissions completed](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/roles_connection_permissions.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=06bb969329f3050b73fea6a363d6a3f5) > [!NOTE] > > Use caution when defining connection permissions. When a policy grants access to only a subset of connections, it is possible for a user governed by that policy to create a connection that falls outside their scope. When this happens, the user cannot view or list the new connection after its creation, because it is not one of the connections included in the policy. ### Certificate Permissions The **Certificate Permissions** section gives you control over what users can do with public and private certificates. Check the checkboxes next to the **Allowed Certificate Actions** you want to assign, then use **Certificates Access** to assign the policy to one or more certificates. ![Certificate policy settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/roles_certificate_policy.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=1b4fff1cc7395c8be3d7f3dbad7d0960) ### Associating Users with Roles 1. To add users to the role, click the **Associated Users** tab. 2. Click **Add Users**, then select the user(s) who can assume this role. 3. When you are finished, click **Add Users**. 4. Use the **Roles** tab to view summary information for each role, including the associated workspaces and users. Hover over the counts to see details. ![Roles tab with associated users and workspaces](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/roles_complete.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=861d41fb4b09447478076bd3f2134bab) > [!NOTE] > > If a certificate policy includes access to a specific subset of certificates, a user who is governed by that policy can create a new certificate but they cannot view or list it in a connector after its creation. This is because the new certificate is not one of the subset of certificates in the policy. ### Role and Policy Example Imagine a scenario where your company has a group of EDI developers. The company has multiple trading partners; we will use two trading partners and two developers for this example. The trading partner flows are broken up into two separate workspaces: PartnerA and PartnerB. Each EDI developer (DevA and DevB) works primarily in their corresponding workspace (DevA = PartnerA), but they also need to be able to view or send files as necessary in the PartnerB workspace. To manage this, you can create the DevA user, create a custom role for DevA, and assign two policies to the role, as shown in the following image. ![Role and policy example with two policies assigned](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/role_policy_example_263.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=fa57364ae004c9e8ac5c9c7cb1b6c67b) * Policy1 gives DevA full access to the PartnerA workspace. * Policy2 gives DevA limited access to the PartnerB workspace, allowing them to view and send files, but not create or delete connectors, or modify flows or settings. ## User Roles Comparison Table This table describes the functionality available in all types of roles. | Action | Admin | Standard | Support | Custom | | - | :-: | :-: | :-: | :-: | | Modify Flows | ✔ | ✔ | | ✔ | | Update Settings | ✔ | ✔ | | ✔ | | Delete Connectors | ✔ | ✔ | | ✔ | | Create Connectors | ✔ | ✔ | | ✔ | | Upload Files | ✔ | ✔ | | ✔ | | Send Files | ✔ | ✔ | ✔ | ✔ | | Receive Files (on a connector's **Transactions** tab) | ✔ | ✔ | ✔ | ✔ | | Delete Files | ✔ | ✔ | | ✔ | | Delete Transactions | ✔ | ✔ | | ✔ | | Re-queue | ✔ | ✔ | ✔ | ✔ | | Invoke Flow API Endpoints | ✔ | ✔ | ✔ | ✔ | | Create, Modify, and Delete Flow API Endpoints | ✔ | ✔ | | ✔ | | Create, Update, and Delete Connections | ✔ | ✔ | | ✔ | | View Connectors and Flows | ✔ | ✔ | ✔ | ✔ [^1] | | View Application and Transaction Logs | ✔ | ✔ | ✔ | ✔ [^1] | | View Worker Pool Analyses | ✔ | ✔ | ✔ | ✔ | | Upload Private Certificate Files to the Profile | ✔ | | | | | Change Profile Settings | ✔ | | | | | View Audit Logs | ✔ | | | | | Add and Manage Users | ✔ | | | | | Start, Cancel, and Remove Worker Pool Analyses | ✔ | | | | [^1]: You must have access to the workspace to perform this action. # Users Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/settings/users How to create and manage users and their access in CData Arc. The first time you launch CData Arc, you are prompted to create the first application user by setting a username and password. The first user defaults to the [Admin role](./user-roles#admin-role). To create and manage additional users, navigate to the [Settings page](./settings-page), then the **Users** tab. This tab includes a table of all users, and includes information about their roles, their user type and status, whether they are temporary users, their [two-factor authentication (2FA)](../2fa) status, and more. ![Users table](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/users_table_264.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=060c615c92d171a1fcc8d9839329e271) Members of the Admin role can create, delete, and modify users on this tab. Admin is the only role with permission to manage other users. ## Add or Edit Users To add a user, click **Add User**. To edit an existing user, hover over their row in the table, click the ellipses, then choose **Edit**. Complete the following fields: | Field | Description | | - | - | | **User Type** | Choose the type of user account: **Standard**, **SSO**, **Active Directory**, **LDAP**, or **Service**. The selected type determines which other fields appear in the form. The available types depend on your license tier. See [User Types](#user-types) for details. | | **Username** | Enter a username. This field cannot be changed once you save the user. For **Active Directory** accounts, use the `domain\username` format. | | **Temporary Account** | If the user should only have temporary access, check this and supply an **Expiration Date**. | | **Email Address** | Best practice is to provide an email address for each user so that they can [reset their passwords](#password-resets) when necessary. | | **Roles** | Select the [user roles](./user-roles) to assign the user. | | **Password** | Enter a password. Ensure that it meets the requirements in [Sign In Settings](./sign-in-settings). The **Password** field appears only for Standard accounts and for SSO accounts with the Admin role. Check **Require password reset on next sign in** to force the user to change their password on login. This checkbox is also available for SSO accounts with the Admin role, which keep a local password. | | **Status** | Indicate whether the user is active or inactive. | | **Federation Id** | For **SSO** accounts, enter the value from your identity provider (IdP) that maps the SSO identity claim to this Arc user account. This field always appears for SSO accounts regardless of whether SSO is globally enabled. For more information, see [Single Sign-On](./single-sign-on). | To give the new user [Admin API](#admin-api-access) and/or OAuth access, click the **API Access** tab. ![Provide Admin or OAuth access](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_user_api_access.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=d871e0cab8a00327b922e0b3292800a4) | Field | Description | | - | - | | **API Access** | Check **Enable Authtoken Access** to grant the user an authtoken that can be used to access the Admin API. See [Admin API Access](#admin-api-access) for details. When you enable API access, the rest of the fields in this table appear. | | **Token Expiration (days)** | The number of days before the token expires. Check **No Expiration** if you do not want the token to expire. | | **Authtoken** | The system-generated authtoken. Make sure to copy it to a safe location before saving the user, because it is not shown again. | | **Maximum Requests per Hour** | Uncheck **Unlimited Requests** to set a limit on the maximum requests per hour for this user. | | **Maximum Concurrent Requests** | Uncheck **Unlimited Requests** to set a limit on the maximum number of requests that can be sent simultaneously. | | **Enable OAuth Access** | Check **Enable OAuth Access** to grant the user credentials that can be used to secure public API endpoints for the [Admin API](./admin-api-settings). OAuth 2.0 provides a structured authentication flow for clients that need to manage token lifecycles or integrate with external applications. When you enable OAuth access, the following fields appear: | | **Grant Type** | The OAuth 2.0 grant type that determines how the client application obtains an access token. **Client Credentials** is currently the only option. | | **Client Id** | The system-generated client Id token. | | **Client Secret** | The system-generated client secret. Make sure to copy it to a safe location before saving the user, because it is not shown again. | | **Access URL** | The endpoint URL used by the client application to request an access token using the configured credentials. | To add trusted or banned IP ranges specific to this user, click the **Trusted IP Ranges** or **Banned IP Ranges** tabs to add the ranges. You can provide single IP addresses, simple ranges (such as `203.0.113.*` or `203.0.113.1-255`), or Classless Inter-Domain Routing (CIDR) ranges. These tabs also display the global trusted and banned ranges that have been defined on the [Network Access](./network-access) portion of the **Settings > Security** page. ## User Types Arc supports the following user types: * **Standard** accounts are for people who sign in to the Arc console interactively with a username and password. * **SSO** accounts are for people who authenticate through an external identity provider (IdP). The **Password** field is hidden for SSO accounts that do not have the Admin role. SSO accounts with the Admin role retain a local password as a fallback for signing in when SSO is unavailable. The **Federation Id** field is always shown for SSO accounts and is used to map the identity claim from the IdP to this Arc user account. Because authentication happens at the IdP, Arc 2FA does not apply to SSO accounts, so the 2FA controls are not shown. For more information on configuring SSO, see [Single Sign-On](./single-sign-on). * **Active Directory** accounts are for people who authenticate using a Windows domain account. Usernames must be in `domain\username` format. Active Directory is only available on the [.NET edition](../../installation/windows-edition). * **LDAP** accounts are for people who authenticate using an LDAP directory service. LDAP is only available on the [Cross-Platform edition](../../installation/cross-platform-edition). * **Service** accounts are for automated, machine-to-machine access, such as invoking [Flow API](../../../flows/flow-api) endpoints or calling the [Admin API](./admin-api-settings). Service accounts do not sign in interactively, so the **Password** field and 2FA controls are not shown; instead, they authenticate only with an authtoken or OAuth client credentials configured on the **API Access** tab. Service accounts require at least one authentication method configured on the **API Access** tab (either **Enable Authtoken Access** or **Enable OAuth Access**). If neither is enabled, the form displays a warning and blocks saving. For example, you might pair a service account with a role that grants only the [Flow API **Invoke** permission](./user-roles#flow-api-configuration) to let it call specific endpoints without any access to, or visibility into, the Arc UI. > [!NOTE] > > When SSO is globally enabled, the **Federation Id** field also appears for Standard accounts. This supports Standard accounts that have a Federation Id but were not migrated to the SSO type, such as Admin accounts that keep a local password as a fallback. The available user types depend on your license tier: | License Tier | Available User Types | | - | - | | Enterprise | Standard, SSO, Active Directory, LDAP, Service | | Professional | Standard, Service | | Standard | Standard | ## Single Sign-On Arc supports single sign-on (SSO) via identity providers that implement the OpenID standard. For more information, including specific guidance for Azure AD, see [Single Sign-On](./single-sign-on). ## Admin API Access Each user is granted an authtoken that can be used to access the [Admin API](./admin-api-settings). For more information on authenticating against the Admin API, see [Admin API Authentication](../../../admin-api/admin-api-authentication). The specific actions that a user can perform using the Admin API mirror the actions that the same user can perform using the UI. For example, a user that cannot delete connectors in the UI cannot use the Admin API to delete connectors. To perform any action using the Admin API, use an authtoken from an Admin user when invoking the API. ## Password Resets Every user can have an email address associated with their account. This way, if they need to reset their password, they can click the **Forgot password** link to have a reset link emailed to them. Every password reset that is sent adds a row to the [application log](../activity#application-logs). Users who do not have an email associated with their account must contact an administrator to have their password reset, so it is best practice for administrators to associate an email with each user account. If all admin users are locked out of Arc, the embedded web servers in each edition provide the ability to reset their password to regain access to the application. This requires administrative access to the installation, and should only be used if there is no user with access available. For example, in the [Cross-Platform edition](../../installation/cross-platform-edition): ``` java -jar arc.jar -ResetPassword -User <user> -Password <password> -AppDirectory <appDirectory> ``` In the [.NET edition](../../installation/windows-edition): ``` CData.Arc.exe -ResetPassword -User <user> -Password <password> -AppDirectory <appDirectory> ``` # Split-Plane Deployment Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/administration/split-plane-deployment How to host CData Arc's Administration Console and public receiving endpoints on separate network ports for enhanced security. By default, CData Arc hosts the Administration Console (where administrators create and manage flows) and the public receiving endpoints on the same network port. To enhance security, you might want to separate the console from the public endpoints so that they are hosted on different ports. This configuration is referred to as a split-plane deployment. Once the console and endpoints are separated, external partners that connect to Arc public endpoints cannot access the Administration Console *even if they acquire the login credentials*. (The default configuration protects the console behind a username/login combination). ## Overview Separating the console from the public endpoints requires explicitly configuring a second web application that consists of only the resources required to host public receiving endpoints. After this additional configuration, Arc uses two ports: one for the full (console) application, and one for the endpoint-only application. This topic describes configuring the endpoint-only application. To configure the full application, see the standard web-configuration process that is described in the [Installation and Configuration](../installation/installation-configuration) topics. ## Restrictions There are two requirements for separating the Arc console and public endpoints: * Hosting Arc public endpoints via an external server (Microsoft Internet Information Services \[IIS], Eclipse Jetty, Apache Tomcat, and so on) * Configuring an enterprise application database (MySQL, SQL Server, PostgreSQL) The following sections provide more details about these requirements. ## .NET Edition ### Setting Up the Endpoint-Only Application If you use the .NET edition, you must use IIS to host the public receiving endpoints. You can still use the embedded web server to host the full (console) application that is not exposed to the public. The .NET edition includes a `www_services` folder that contains the web configuration data required to host an endpoint-only application. When IIS is directed to this folder, the Arc public endpoint is accessible through IIS, without exposing the Administration Console. Further configuration of ports, SSL, and so on should be accomplished using IIS directly. ### Setting Up the Application Directory and Application Database Separating the public endpoints from the console requires configuring an enterprise database application. The `www_services` folder includes a `web.config` file where you configure the application directory and application database connections. When you configure a separate installation for the services endpoint and administration console, CData strongly recommends that both configurations point to the same application directory and database. For details about these configurations, see [Configuring the Application Directory](../installation/windows-edition#configuring-the-application-directory) and [Configuring the Application Database](../installation/windows-edition#configuring-the-application-database). > [!NOTE] > > The `www_services` folder is separate from the `www` folder, which hosts the full (console) application. The configuration of the `www_services\web.config` file should match the configuration of the file with the same name in the `www` folder. CData recommends that you manually update the `\www_services\web.config` file instead of copying the `web.config` file from the `www` folder because there is web configuration related to the user interface that is not needed in the services endpoint. ## Cross-Platform Edition ### Setting Up the Endpoint-Only Application For [Cross-Platform](../installation/cross-platform-edition) installations, you can use the embedded web server to host either the full console application or the endpoint-only application. To set up the endpoint-only application, generate the `services.war` file that is used when you separate the endpoints from the console. Run the following command (this command also generates the `services.properties` file). This outputs the `services.war` file to the `webapp` folder and the `services.properties` file to the installation directory (where `arc.jar` is located). ``` java -jar arc.jar -CreateServicesWAR ``` The `services.properties` file includes the contents of the `arc.properties` file and overrides the WAR path and ports, as shown below: ``` !include arc.properties cdata.warPath=./webapp/services.war cdata.stopPort=18753 cdata.http.port=8088 ``` After you have run `CreateServicesWAR`, start the services endpoint by running: ``` java -jar arc.jar -config services ``` You can also use the `-config` parameter to point Arc to a configuration file other than the default `arc.properties` file. See [Configuring the Embedded Jetty Server](../installation/cross-platform-edition#configuring-the-embedded-jetty-server) for details. ### Setting Up the Application Database Separating the public endpoints from the console requires configuring an enterprise database application. For details about configuring the enterprise application database, see [Configuring the Application Database](../installation/cross-platform-edition#configure-the-application-database). # Architecture Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/architecture Design principles, core concepts, and implementation details of CData Arc's message-driven integration platform. CData Arc is a message-driven platform for integrating business processes. It provides a centralized communications hub that enables applications, databases, and external messaging systems to talk to each other. ## Design Principles and Implementation The Arc architectural decisions flow from a set of core design principles. This section focuses on three aspects of Arc design: * Principles and values that we prioritize * Core concepts necessary for a high-level understanding of Arc * Architectural decisions and how they fit with our design principles The [Core Principles](#core-principles) and [Concepts](#concepts) sections are intended for both technical and non-technical users. The remaining implementation sections are intended to help engineers understand how to leverage the Arc architecture. ### Core Principles The following diagram illustrates the core principles of Arc design: ![Core principles diagram](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_core_principles.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=c86cf0205740495242cd9d2105743836) The Arc architecture is not the only way that the design principles above could be implemented in an integration platform. What follows are explanations of our implementation choices and how they fit with our design principles. ## Concepts At a high level, Arc implementations are composed of three elements: * Flows * Connectors * Messages The following diagram illustrates the relationships between these elements: ![Data processing relationships](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_data_processing.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=d27aec49fee4da70c21e9929aa14a277) ### Flows Arc [flows](../flows) are configured workflows that perform an automated set of data processing tasks. Flows are composed of a number of connectors that are connected to each other so that data processed by one connector is automatically passed along to the next connector in the flow. In the UI, flows are held in workspaces on the **Flows** page. Connectors are added to the blank canvas where they can be configured and connected to each other. The image below represents a simple flow consisting of two connectors: AS2 and File. ![Simple flow with two connectors](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_flow.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=623db5d096067bfa100412b4bc150945) The blue arrow between the connectors indicates that data received by the AS2 connector is automatically passed along to the File connector. Dragging the blue dot on the right side of a connector onto the left side of another connector establishes this relationship. The series of connectors in a flow establishes a logical chain of data processing, so flows should be designed to accomplish a specific business task. These tasks can vary in complexity from simply downloading a file all the way to synchronizing multiple back-end applications in accordance with received business documents. You can configure multiple flows in Arc to accomplish separate tasks. Flows act independently of each other unless they are explicitly connected to each other by a blue flow arrow. ### Connectors [Connectors](../connectors) are the individual building blocks of a flow, operating on application data, or messages. Each connector performs a specific operation on a message and passes it along to the next connector in the flow, or to an outside party (such as an FTP destination). Operations fall primarily into three categories: Trigger, Transform and Terminal. * Trigger connectors start a flow by **Receive Automation** pulling or creating a file or, for connectors that support passive receive, when the connector receives a file from an external client. Examples include [AS2](../connectors/as2), [Email Receive](../connectors/email-receive), or [SFTP](../connectors/sftp). * Transform connectors are connectors that reside in the middle of a flow and are responsible for performing work on the message (modifying it in some way). Examples include [X12](../connectors/x12), [XML Map](../connectors/xml-map/xml-map), or [CSV](../connectors/csv). * Terminal connectors are the end of a flow, where Arc hands the message off to the underlying disk or where the message is consumed by the connector during its send action and no output is generated. Examples include [File](../connectors/file), [MySQL](../connectors/mysql/mysql), or any database connector configured for Upsert. Each time a connector performs one of the above operations it is processing a *transaction*. In the workspaces on the **Flows** page, when you add a connector to a canvas its configuration settings panel opens: ![Connector configuration panel](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_connector.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=dcbc712c04ec3f1b0ed44863814545a6) Connectors have different sets of configuration settings based on the connector type (AS2, SQL Server, XML, and so on). Some connectors, such as AS2 and SFTP, either receive data from an external party or server or send data out to an external party or server (or both). These connectors are typically at the start or the end of a flow, since that's where data enters or exits the application. Other connectors, such as XML Map and X12, process data locally in the application. These connectors are typically in the middle of a flow, since they can only receive data from other connectors (not from an external source) and can only send data to other connectors (not to an external destination). ### Messages When data is passed between connectors in a flow, it is passed in the form of a message. Messages consist primarily of file payload data (the data that Arc is processing) and metadata (information that Arc uses to track the flow of data through the application). Messages are explained in more detail in [Message Implementation](#message-implementation), but the high-level takeaway is that messages are files in standard RFC822 (Internet Message) format. When a connector passes data to another connector in the flow, it writes the data to a file then passes that file to the next connector down the line, as illustrated in the following diagram. ![Message flow between connectors](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_message_flow.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=44efec8e6673b343000367184efe4d8b) At each step in a flow, a message (file) is received or downloaded into the application (using a Trigger connector), transformed locally in the application (using a Transform connector), or sent or uploaded out of the application (using a Terminal connector). Each of these operations is referred to as a *transaction*. Arc is transparent: you can observe what happens inside the platform by looking at the file system, seeing how messages are copied from one folder to another, and viewing message content using a regular text editor or other tool. ## Implementation Overview This section provides technical details about Arc's implementation of the core concepts defined above ([Flows](#flows), [Connectors](#connectors), and [Messages](#messages)), together with explanations of why we think our implementation satisfies our [Design Principles](#core-principles). #### File-based Architecture Arc stores all data in files on the filesystem, so everything in the application persists to disk. For example: * Application data (messages) * Configuration data ([profiles](./administration/profiles) and flows) * Logs (see [Activity](./administration/activity) for details) Connectors in a configured flow read and write files (messages) from the **Transactions** tab. When data passes from one connector to the next, these message files are moved from the Receive folder of the first connector into the Send folder of the next connector. Arc provides a web UI on top of this filesystem-based infrastructure to easily construct and configure flows. Arc's [Admin API](../admin-api) and UI are the recommended methods for managing configurations. These methods make direct modifications to the configuration files on disk. The file-based architecture also makes it easy to integrate Arc with Infrastructure as Code (IaC) tools and version control systems like Git. For more details on how to integrate Git with Arc, see [Using Git with CData Arc](https://arc.cdata.com/kb/articles/arc-git.rst). ## Message Implementation Messages in Arc are simple files on disk that contain the raw data processed by the application. Messages have two parts: * Body: the application data payload * Headers: message metadata The combination of headers and body is stored in RFC2822-compliant files with an `.eml` extension that can be opened by any standard text editor or email client. The headers are a simple list of `name: value` headers separated by `newline` characters, and the body is separated from the headers by two `newline` characters. > [!WARNING] > > To ensure that Arc performs reliably, do not interact directly with these files while the application is running. External programs placing locks on these files can cause performance and reliability issues. ### Message Body The message body or payload is the actual file data being processed by the application. This is the data that is received from a remote source, manipulated by transformation connectors, and so on. While the message headers are primarily used by Arc for tracking messages, the message payload contains the data that users care primarily about. When data is received, Arc generates a message (it writes a new message file to disk) and uses the incoming data as the message body. For example, when an SFTP connector downloads a file from a remote server, the contents of that remote file become the body of a new message. When data is processed locally in the application (such as when EDI is translated into XML, or when a file is mapped into a new format), the connector performing the operation reads the body of the input message, processes the data in memory, and writes the results into the body of an output message. When data leaves the application (for example, when uploading a file to a remote server, sending a file to a peer, or inserting into a database), only the body of the input message is sent out. ### Message Headers Message headers help Arc track the progress of data through the application. Headers include a unique MessageId (which helps Arc know the full lifecycle of a message, even if the filename is changed), timestamps from when connectors processed the message, any errors that might have occurred during processing, and other metadata. Headers are listed in *headerName: headerValue* syntax at the top of the message, delimited by line breaks. Hovering over a message and clicking **View Details** shows its associated headers, and lets you download the message file content. Message headers are also used for miscellaneous values that are helpful for Arc to know in a flow. For example, when downloading files from a subfolder on a remote FTP server, Arc uses a [subfolder header](../connectors/ftp#subfolder-headers) to track the folder path to the message in case this folder path needs to be recreated on the local system. Arc augments the raw data files with metadata that describe how the application has processed the file. These are some of the most common headers: * A unique, durable MessageId that identifies the file throughout the flow * Timestamps of the file being processed * The ConnectorId of any connector that processed the file * Any instance of the file failing to process due to an error Message headers are never edited or overwritten by the application. New headers are always appended to the existing list so that the full processing history is preserved. ### Messages in a Flow While Arc uses message headers internally for tracking and understanding the data processed by the application, it hides these details from users unless the message is inspected. For example, Arc uses an internal MessageId to identify a message, but displays a public filename in the **Transactions** tab, and on the **Messages** tab. To examine the headers on a message, hover over the message and choose **View Message Details**. The following image shows messages on the **Transactions** tab. ![Messages on the Transactions tab](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_headers.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=f6c1a822fe909b36ea9486fa89e3555c) Arc stores messages in the `.eml` format when they are written to a connector's internal Receive folder, regardless of where the message is in the flow. However, when messages are sent to an external system (for example, a database or an SFTP server), Arc only uploads the file content and does not include the `.eml` headers. Similarly, when writing files to disk using the [File connector's](../connectors/file) Send action, Arc omits the `.eml` structure so that the output matches what external processes expect. In other words, `.eml` headers are part of Arc's internal file format but are not included when delivering data to external destinations. ### Batch Groups and Batch Messages Messages can contain multiple payloads, in which case they are considered *batch groups*. Each individual payload in a batch group is considered a *batch message*, as illustrated in the following diagram. ![Batch groups and batch messages](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_batch_processing.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=0cfc0305fa14b3cc0a956e7c71957c00) #### Batch Groups Messages can be batched together into batch groups to improve performance and make large groups of messages more manageable. Batch groups are MIME-format files where each MIME part is a separate batch message. The batch group maintains metadata about the batch using the same header scheme that basic messages use. These headers track the processing of the batch as a whole, rather than individual parts of the batch. #### Batch Messages Each batch message in a batch group is a MIME part containing the file payload data processed by the application. Each batch message contains metadata associated with the payload (but not the batch group) in the MIME part. Batch messages have multiple sets of metadata: one set of headers for tracking the batch group as a whole, and a separate set of headers for each message in the batch. ### Accessing and Viewing Messages Because messages are just simple files on disk, users and external systems can access and view messages currently in the Arc pipeline. To access messages with an external application, you need to understand where on the filesystem to look for the message `.eml` files. See [Folder Hierarchy](#folder-hierarchy) to learn the exact location of the message files. > [!WARNING] > > To ensure that Arc performs reliably, do not interact directly with message files while the application is running. External programs placing locks on these files can cause performance and reliability issues. You can also see messages in the Arc UI and the Arc [Admin API](../admin-api). Messages are displayed in the **Transactions** tab for any connector that has processed a message, as well as on the [Activity](./administration/activity) page. Click **View Details** to open the **Message Information** page: ![Message Information panel showing headers](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_message_headers.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=1d5ab8427804480e6e30ec89c82ddc6d) The values displayed in this panel are the message's headers. Click the **Download** button to access the message body. Click **Download Logs** to download the [logs](#message-tracking-and-logs) associated with a message. ### Message Tracking and Logs Messages include a MessageId header that functions as a unique identifier for messages. This MessageId remains the same as the file passes through the flow, even if the file itself is renamed. As a result, the progress of any file throughout a flow can be traced by referencing its MessageId. This value also allows users to find log data associated with a particular transaction. Arc stores log data in two formats: * [Application database](#application-database) * [Verbose log files](#verbose-log-files) on disk These formats and the relationship between the two are detailed below. #### Application Database The *application database* is a relational database that Arc uses to: * Store metadata about each transaction processed by the application * Store logs for application operations that are not specific to any transaction * Store the state of certain connectors (for example, when a connector is awaiting an acknowledgment or receipt) Arc comes packaged with a SQLite (Windows/.NET) or H2 (Java) database, but it can be configured to use an external database like SQL Server, PostgreSQL, or MySQL instead. Arc handles the creation and maintenance of all relevant tables in the application database. To use an external database, you must configure the database connection string in the configuration file for the server that hosts the application (see [Configuring the Application Database](./installation/windows-edition#configuring-the-application-database) in the [.NET edition](./installation/windows-edition) and the [cdataappdb](./installation/arc-properties-configuration#cdata-app-db) setting in the `arc.properties` file in the [Cross-Platform edition](./installation/cross-platform-edition)). There is a table in the application database that is used to track messages and find logs. It stores metadata for each transaction processed by the application, and the metadata includes the MessageId of the message that was processed. Use the **Activity** page in the UI to find messages and transaction logs. You can search for transaction metadata (such as timestamp, ConnectorId, and status) on the **Activity** page. To minimize the database footprint, verbose logging data is not included on the **Activity** page and is instead stored in log files on disk. #### Verbose Log Files Each time Arc processes a message, it generates a log folder with the same name as the MessageId. This folder contains the verbose log files, and the specific logging content depends upon the type of connector that sent and/or received the file. For example, the AS2 connector logs the MDN receipt, raw request and response, and connection log in addition to the `.eml` file representing the message itself. These log files are not necessary for merely tracking a message through the flow; the metadata stored in the application database is enough to track a message. However, to find detailed information on a specific transaction (especially when debugging), you need to find the appropriate files on disk based on the MessageId of the message and the connector that processed it. Users can manually navigate to the appropriate log folder on disk once they know the MessageId; however, it is often more convenient to use the application database to find these verbose log files. See [Relationship Between Application Database and Log Files](#relationship-between-application-database-and-log-files) for details. #### Relationship Between Application Database and Log Files The metadata stored in the application database includes everything needed to find the log files associated with a particular transaction. The Arc UI and [Admin API](../admin-api) take advantage of this relationship to provide easy access to the log files for any transaction. * The **Activity** page contains all the transaction metadata. Hover over any transaction entry and click **View Details** to access the associated log files; underneath, Arc is using the MessageId to find the appropriate log files on disk. * Similarly, the **Transactions** tabs on connector configuration panels provide access to any transaction processed by that specific connector. Expand these transactions to view and download verbose log files. * The application database can also be helpful for manually navigating to the appropriate folder where transaction log files are held. Often, a user knows the filename of a transaction but not the internal MessageId. The **Messages** tab on the **Activity** page is searchable, so searching for a specific filename brings up the MessageId for any transactions associated with that filename. See [Folder Hierarchy](#folder-hierarchy) for more information on how to find the log folders once you know the MessageId and connector. ### Design Decisions Behind Message Implementation We want application data to remain transparent to users and external systems, so messages are passed using simple data files on the filesystem rather than an opaque internal data channel. As a result, Arc is not a black box where data is hidden away during processing. This means that Arc can effectively be embedded into any system that picks up or drops off files into specific folders on a filesystem. We also want messages to be accessible without specialized tools or programs. Messages are stored in RFC2822-compliant files so that any text editor can open the file. Because messages have a `.eml` extension, double-clicking on a file opens the message and displays its contents in the system's default email client. So that messages and logs are accessible and transparent, we provide a searchable database of transaction metadata. This database can be used to access the message payload and verbose log files directly, while maintaining a light database footprint. ## Connector Implementation Connectors represent a single step in a [flow](../flows). Complex flows are created by connecting sets of connectors together to perform a logical chain of data processing. Connectors perform three types of operations: * Receive data and write an output message (such as downloading a remote file over SFTP) * Read an input message and send the data to an external party (such as sending a file over AS2) * Read an input message, process it locally, and write an output message (such as translating an EDI file into XML) Most connectors can perform either the first two operations (send and receive data to/from remote endpoints) or the last operation (transform data locally). Every operation performed by a connector is referred to as a *transaction*. Connectors read and write input and output messages from the **Transactions** tab. When connectors are connected in the flow, Arc automatically moves messages from the **Transactions** tab of the first connector to the **Transactions** folder for the next connector. (On disk, the input folder is labeled Send, and the output folder is labeled Receive.) ### Connector Files and Folders Every instance of a connector exists on disk as a single folder named for the ConnectorId (the display name in the flow). Each connector folder contains the following: * A [port.cfg](#port-cfg) file containing configuration settings * Messages in the [Send](#input-messages) subfolder to send or process (such as input messages) * Received messages in the [Receive](#output-messages) subfolder (such as output messages) * [Log files](#log-files) for messages processed by the connector * [Connector-specific files](#additional-configuration-files) like maps, templates, and scripts The following sections explain how these folder contents function in the application. [Folder Hierarchy](#folder-hierarchy) explains the exact location of connector folders and subfolders. #### Port.cfg Each connector's `port.cfg` file contains the settings that govern the connector's behavior (the settings that appear in the connector's **Settings** and **Advanced** tabs). This file is in INI format: connector settings are listed in `SettingName = SettingValue` syntax, one-setting-per-line. Editing connector settings in the UI is functionally equivalent to editing the settings directly in the `port.cfg` file. The `port.cfg` file supports *indirect values*, which means settings can use a reference instead of a string literal. This is conceptually similar to setting a variable in other sections of the application, and referencing the variable name in the `port.cfg` file. See [Data Directory](#data-directory) for details on how to set indirect values. #### Input Messages The **Transactions** folder holds input messages, or messages that are queued to be processed by the connector. At every clock tick (half a second by default), Arc checks for changes in each connector's Send folder and dispatches worker threads to any connector with available messages. When a message is processed, Arc appends a header to that message with the timestamp of the processing attempt. Arc sorts the input messages according to the last modified time so that older files are processed first. Since Arc appends a header during each attempt, this ensures that messages that cause an error do not block the connector unnecessarily through repeated retries. Note that connectors must have *Send* automation enabled in order to process input messages every clock tick. #### Output Messages The Receive folder holds output messages, or messages that have been received, downloaded, and/or processed by the connector. Some connectors write an output message as soon as they finish processing an input message (such as connectors that translate data formats). Other connectors write an output message without first reading an input message. Examples include an SFTP connector that polls a remote server for files to download, and an AS2 connector that passively listens for incoming files. When a connector is connected to another connector in the flow, output messages do not remain in the Receive folder. They are passed along to the Send folder for the next connector. #### Log Files Each transaction processed by a connector generates a set of log files. Metadata about the transaction is added to the log, and the [verbose logging](#verbose-log-files) information is stored as files on disk. The log files always include the message itself in `.eml` format along with any connector-specific log files. By default, all connector log files are organized in folders by week, as shown in the following folder structure: ![Log folder hierarchy](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_logs_folder_hierarchy.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=23c2294bce0cf1ba77e90c2915067975) To change the default weekly folder organization, set the **Log Subfolder Scheme** field of any connector to a different time interval (such as *Daily* or *Monthly*). This means that all logs generated in that time interval are held in one subfolder. This can improve disk I/O performance by reducing the size of any individual subfolder. You can use the [Messages tab](./administration/activity#messages-tab) to find the appropriate log folder, either by finding the MessageId or by using **Download Logs**. #### Retaining Copies of Processed Files The recommended way to retain copies of successfully processed files is to right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../connectors/file). The File connector writes the processed payload to a local directory of your choice. Files routed through the Success Path are covered by Arc's Encryption at Rest while in transit through the application; once written to the File connector's target directory, they are stored as plain text. > [!NOTE] > > The **Save to Sent Folder** setting, which copied processed files to a Sent subfolder inside each connector's folder, is deprecated as of v26.3 and disabled by default. It remains visible and functional on connectors where it was previously enabled, but is not recommended for new configurations. Files in the Sent folder are not covered by Encryption at Rest. #### Additional Configuration Files Depending on the connector type, some connector folders include additional configuration files. These additional files include: * `map.json` * `script.rsb` * template XML files These files store mapping relationships, custom scripting behavior, and the structure of connector input and output. Manually editing these files is equivalent to editing the associated connector UI settings. > [!WARNING] > > It can be risky to manually edit any of these files. The layout is easy to understand and transparent, but incorrectly modifying these configuration files can have unintended consequences. ### Design Decisions Behind Connector Implementation We want flows to be fully modular, which means that each connector should provide a straightforward function in a predictable and intuitive way. Arc's capacity to perform complex tasks comes from the ability to combine arbitrary sets of connectors, while the connector interfaces remain simple. Sophisticated flows remain easy to understand because the connector chain is broken down into discrete steps. We want all of the data relevant to a connector (configuration data, application data, logging data, and so on) to be easy to access. Because the data is all stored in a connector-specific folder, accessing the data is just a matter of knowing where to find the folder. Configuration data is stored in the transparent INI format so that it, like message data, can be easily viewed and edited with simple tools. Our folder-based approach to connectors also makes it easy to understand how data is passed between connectors: a simple file-move operation on the message files turns one connector's output into another connector's input. Arc includes built-in tools to establish these file-move relationships automatically. > [!WARNING] > > To ensure that Arc performs reliably, do not interact directly with any configuration files while the application is running. External programs placing locks on these files can cause performance and reliability issues. ## Flows and Workspaces Implementation [Flows](../flows) are represented on the filesystem in a `flow.json` file that contains the position and connections for each connector in a flow. See [Folder Hierarchy](#folder-hierarchy) for the location of this file. The position of connectors in a flow is purely cosmetic and is only used when you configure flows in the UI. The connections between connectors are relevant at the data processing level: after a connector writes an output message, it queries the application engine to see if it needs to pass the output message to another connector's Send folder. The application uses the `flow.json` file to determine which connector, if any, is given the file. You can configure multiple flows on the same flow canvas, and the Arc UI lets you drag the canvas around to navigate between flows. This is similar to navigating geographical maps such as Google Maps online. [Workspaces](../flows/workspaces) provide a logical separation between flows. Each workspace provides a fresh canvas within which to configure connectors into logical flows. Extending the Google Maps analogy, workspaces function as separate *planets*. It is common practice to separate workspaces by trading partners or integrations. However, deciding whether to configure new flows in separate workspaces is a matter of preference. See [Organize Flows into Workspaces](../flows/designing-a-flow#organize-flows-into-workspaces) for more information. #### Workspaces on the Filesystem When you introduce new (non-default) workspaces, Arc adds a new set of folders in the folder hierarchy. The `workspaces` directory contains the connector folders for all connectors configured in non-default workspaces. The `workspaces` folder is a sibling of the [Data Directory](#data-directory). ### Design Decisions Behind Flows and Workspaces We think that the best way to preserve the simple and modular relationship between connectors in a flow is to treat flows as a purely logical concept. All of the information necessary to determine the logical series of data processing is already contained in the connector implementation. We want workspaces to provide the ability to separate connectors on a more substantial level. Separate workspaces use separate folders to hold connectors so that filesystem-level operations (such as permissions and directory listings) can be easily isolated to a particular workspace. ## Folder Hierarchy Because all Arc resources are available on the filesystem, understanding Arc at a low level requires learning the application's folder structure and how files and folders are organized on disk. To understand the hierarchy, it is helpful to see the directory view of an Arc installation. Following is a visual representation of the folder structure. The details of this structure are explained in the following sections. ![Folder hierarchy](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_folder_hierarchy.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=27941bcd0a3c2ad03baf6cfb618c7772) > [!NOTE] > > * This hierarchy assumes that flows are only configured in the default workspace. See [Flows and Workspaces](#flows-and-workspaces-implementation) for details on additional workspaces. > * If **Local Archive Folder** in the [Cleanup Settings](./administration/settings/cleanup-settings) section of the **Settings > Advanced** page is empty, each connector also has an **Archive** folder. ### Root Installation Directory All Arc resources are in the installation directory, in the following default locations: * **Windows**: `C:\Program Files\CData\CData Arc` * **Linux**: `/opt/arc` For Java installations, CData provides a zip file that you can extract to the installation directory of your choice (for example, `/opt/arc`). When the Java edition of Arc is hosted on an external Java servlet like Tomcat, Arc resources reside in `~/cdata/arc`. In this path, `~` resolves to the home directory of the user that hosts the Java servlet. In the [folder hierarchy](#folder-hierarchy) above, the root installation directory is the `Arc` folder at the top of the tree. ### Data Directory The `data` directory contains the following: * A subfolder for each connector configured on the [flows](../flows) page (in the default workspace) * A [profile.cfg](#profile-cfg) file * A [flow.json](#flow-json) file * A [roles.json](#roles-json) file * A [settings.json](#settings-json) file * A [users.json](#users-json) file * All [certificate](#certificates) files #### Profile.cfg The `profile.cfg` file contains application-wide settings such as those for configured local [profiles](./administration/profiles) (for example, the AS2 profile). This file is in INI format: application settings are listed in `SettingName = SettingValue` syntax, one-setting-per-line. The `profile.cfg` settings are divided into sections: an `Application` section for general application settings, and a dedicated section for each profile (for example, there is an `AS2` section for the AS2 profile settings and an `SFTPServer` section for the local SFTP Server settings). Editing application settings on the **Profile** page of the UI is functionally equivalent to editing the settings directly in the `profile.cfg` file. #### Flow\.json The `flow.json` file contains the structure of configured flows: the positions and connections between each connector instance. Editing `flow.json` directly accomplishes the same thing as moving or re-connecting connectors on the **Flows** page. #### Roles.json The `roles.json` file includes details related to all custom [user roles](./administration/settings/user-roles). Editing `roles.json` directly accomplishes the same thing as adding and configuring roles on the **Roles** page. #### Settings.json The `settings.json` file contains a list of settings that can be referenced elsewhere in the application by using reference names instead of values. This enables you to store secure values (like passwords) in a way that does not display them in the application. You can also use reference names to centralize the configuration of flows that are deployed across multiple instances. Settings that support this centralized reference syntax have a key icon in the UI, which you can click to view a list of defined references in the `settings.json` file. For more information, see [Global Settings Vault](./administration/settings/global-settings-vault). #### Users.json The `users.json` file contains details related to all Arc [users](./administration/settings/users). Editing `users.json` directly accomplishes the same thing as adding or editing users on the **Users** page. #### Certificates All certificates created or uploaded in the application are stored in the `data` directory. Typically, certificates come in public/private pairs with the same filename but different file extensions: `.pfx` for private and `.cer` for public. Certificates in other formats that are uploaded to the application are stored as-is. Arc enumerates the certificates in the `data` directory to determine the options available when configuring a certificate in the UI. Adding a certificate file to the `data` directory accomplishes the same thing as uploading a certificate in the UI. ### Connector Folders Each configured connector instance has a dedicated folder in the `data` directory. The folder names match the ConnectorId value (the display name) for each connector in the flow. In the example folder structure above, the two configured connectors are AS2\_Amazon and Database\_MySQL. The contents of the connector folder are explained in [Connector Files and Folders](#connector-files-and-folders). Following is an overview of the subfolder structure in the connector folder: * Logs: Verbose log files for each message processed by the connector * Received: Logs for messages received by the connector; each message has its own folder matching the MessageId * Sent: Logs for messages sent by the connector; each message has its own folder matching the MessageId * Receive: Messages that the connector has received or processed * Send: Messages that the connector should send or process * Sent: Copies of successfully processed messages (in raw data file format). Only present when the deprecated **Save to Sent Folder** setting is enabled. Depending on the connector type, some connectors can have other subfolders: * Pending: Processed messages that are pending an acknowledgement from another party * Public: Files that should be published as public endpoints in the application * Schemas: Schema files (like EDI schemas) that are specific to a particular connector * Templates: XML representations of connector input and output data ### Design Decisions Behind the Folder Hierarchy Because we want the data in Arc to be transparent and accessible, specific files in the application should be easy to find. A simple hierarchical folder structure helps ensure that users and external systems know where to look for the Arc data files. We want the Arc UI to provide a convenient interface for configuring and using the application, but we also want the application to remain fully embeddable in other solutions. Understanding the Arc folder structure is all that is required to point these other solutions to any relevant Arc data. ## Automation The Arc automation service processes files in the **Transactions** folder for each connector. The automation service runs on a half-second *clock tick*, and messages move one step further through the flow at each tick. Arc supports parallel processing by distributing multiple threads to each connector, and each thread can process multiple files in that connector. The specific numbers of workers distributed and files processed per-worker are determined by performance settings in the [profile](./administration/profiles) and in the specific connector settings. ### Clock Ticks At a predefined interval (500 milliseconds by default), Arc checks the contents of the Send folder for every configured connector, and if new files are found, assigns worker threads to process them according to the connector's settings. The application does not guarantee which connector's input is enumerated first. When multiple files are in a single connector's Send folder, they are processed in order according to the last modified time (files modified least recently are processed first). When Arc attempts to process a file, it adds a message header with the processing timestamp, which changes the last modified time for the file. This means that if a file fails to process (causes an error), it becomes the lowest priority file compared to the rest of the files in the Send folder. This prevents a single file from blocking the operation of a connector by repeatedly throwing an error and immediately being retried. ### Parallel Processing When parallel processing is enabled, Arc can distribute multiple worker threads to the same connector during a single clock tick. The number and behavior of these workers are determined by three settings on the **Advanced** tab of the **Settings** page (some of which can be overridden for individual connectors in that connector's **Advanced** tab): * **Worker Pool**: Establishes the global maximum number of threads that the application can assign to all connectors (combined) at the same time. When a thread finishes its assignment in a specific connector, it is recycled back into the pool. The hardware resources available on the host machine determine the upper limit for this setting. * **Max Workers per Connector**: Establishes the maximum number of threads assigned to a single connector at the same time. Each thread assigned to a connector processes files in that connector's Send folder one at a time until the folder is empty (or until the **Max File per Connector** is reached). Once this condition is reached, threads assigned to the connector are recycled back into the worker pool as they finish processing. You can override this setting on an individual connector's **Advanced** tab. * **Max Files per Connector**: Establishes a limit on the number of files processed in a single connector within a single clock tick. For example, if this setting is set to 5, then threads assigned to a connector are recycled back into the worker pool after they have (collectively) processed 5 files, even if there are still files in the connector's Send folder. You can override this setting on an individual connector's **Advanced** tab. When this is set to 0, the connector attempts to process all the files that are present in the connector at the time that the threads are assigned. Threads are distributed to connectors in round-robin fashion, so adjusting these values (either globally or for specific connectors) can help alleviate throughput problems or prevent some connectors from hogging system resources. The following diagram illustrates how the parallel processing settings work together. ![Parallel processing settings diagram](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/architecture_parallel_processing.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=24536542dcff100247264b02bde5d368) #### Maximizing Performance The **Max Workers per Connector** and **Max Files per Connector** settings can be adjusted in individual connectors to help maximize Arc performance when certain connectors require more or less resources than others. Increasing **Max Workers per Connector** for a particular connector tells the application to assign more system resources when processing that connector's input files. This is useful when a particular connector acts as a bottleneck for flow throughput. However, because threads are assigned in round-robin fashion, increasing this setting for many different connectors might cause the application to run out of threads in the **Worker Pool**. When this happens, the application must wait for other threads to finish and be recycled before it can assign them to the remaining connectors. Decreasing **Max Workers per Connector** for a particular connector prevents that connector from severely depleting the worker pool when processing files. This can help avoid issues where the application runs out of threads to distribute during round-robin assignment. However, if the connector's Send folder has many files, it can take a long time for the smaller number of threads to finish processing the files and be recycled back into the pool (unless **Max Files per Connector** is also adjusted, as discussed below). Increasing **Max Files per Connector** for a particular connector (or setting it to `0` to process all files) helps ensure that files do not sit in the connector's Send folder for multiple clock ticks. This can help increase throughput for high-volume connectors. However, it might also increase the chance that threads assigned to the connector are not recycled back into the worker pool for a long time (which could potentially cause a thread shortage). Decreasing **Max Files per Connector** for a particular connector helps ensure that threads assigned to that connector are recycled back into the pool in a timely manner. However, it can mean that files are not always processed within a single clock tick if the number of files in the Send folder exceeds the maximum processed per-tick. Maximizing performance is specific to your particular environment and use case. In general, connectors that need to process or send a large number of files should be assigned more workers and those workers should be allowed to process more files before recycling. To prevent the thread pool from being exhausted, other connectors should be assigned fewer workers and those workers should be limited to fewer files. ### Receive Automation In Arc, the automated processing of input files described above is called *Send* automation. Arc also supports *Receive* automation, which describes the automated process of connectors pulling files into a Arc flow according to a scheduled interval. Receive operations fall into two categories: * Downloading files from a remote host/server (such as FTP, SFTP, or S3) * Pulling data from a back-end system (such as a database, ERP system, or CRM) and writing it out as XML Receive automation is always tied to a scheduling interval that determines how often the connector attempts to download files or pull data from an external system. Every clock tick, the automation service checks to see if any connector's receive interval has elapsed. If so, the connector immediately establishes an outbound connection and pulls data according to the connector's settings. Some connectors, like the AS2 connector, passively listen for incoming data to receive. These connectors do not support receive automation because they cannot actively poll an external system for data. ## Putting This Document to Use Getting started in Arc does not require a full understanding of the underlying architecture. However, peeling back the layers of the application provides an important set of benefits: * [Evaluating](#evaluate-arc-technical-characteristics) whether the Arc approach fits more specific technical requirements * [Manipulating](#manipulate-arc-with-confidence) and configuring the application with greater understanding and control * Understanding how to [embed](#embed-arc-in-a-larger-solution) Arc as part of a larger data-processing solution #### Evaluate Arc Technical Characteristics Arc is a lightweight implementation of a message-driven integration framework. An understanding of its underpinnings helps engineers know whether this approach is suitable for their specific business needs. After reading this document, we hope the principles guiding our design decisions are clear, since these principles continue to shape the growth and development of the application. Understanding the *why* behind an application's design helps users make informed decisions about whether it is right for them. #### Manipulate Arc With Confidence The Arc UI provides an accessible interface for manipulating and configuring the application. A better understanding of the underlying architecture helps users perform more advanced work *under the hood*, such as setting permissions on specific folders for granular control over user access. In addition, we believe that a conceptual grasp of flows, connectors, and their relationship with application data can help users quickly and easily configure optimal workflows. We prioritize the ability for users to take advantage of the Arc feature set with minimal configuration overhead. #### Embed Arc in a Larger Solution Once a user knows the Arc folder structure, the process of embedding Arc in another data processing system is simple. Since Arc interfaces with the system through a set of well-defined folders on disk, any application that reads and writes from specified folders can seamlessly pass data to and from Arc. CData recommends using a [File connector](/26.4/self-hosted/en/connectors/file) when interacting with external systems. See [Interacting with the Local File System](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) for configuration details. #### Feel Comfortable with Arc Finally, we believe that engineers have a common desire to understand the mechanisms underlying the systems that they use. Even if none of the benefits listed above are applicable to your implementation, we hope that this document has provided you with an additional layer of comfort and familiarity that helps you as you use Arc to build your own data integration workflows. If you have any questions about the content of this document or need further guidance, please reach out to our dedicated support team at **[arcsupport@cdata.com](mailto:arcsupport@cdata.com)** and we will be happy to assist you. # Glossary Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/glossary Standard terms used in the CData Arc documentation. The following table provides a set of standard terms used in the product's documentation. | Term | Definition | | - | - | | **Admin API** | The Admin API is a REST API built-in to the application that allows for managing, monitoring, and administering the application from afar or via automated processes. Everything that can be accomplished within the CData Arc Administration Console (web UI) can be achieved by a single or series of Admin API calls. | | **Access Log** | The Access Log records any web request that reaches Arc. This can be helpful in troubleshooting connectivity issues when partners send AS2 or AS4 requests to your web server. The Access Log can be viewed on the Status page of the application. | | **Application Directory** | The Application Directory is a folder that contains configuration files, certificate files, raw data files, and logfiles for the application. For Windows installations, this is `C:\ProgramData\CData\Arc\`. All **Profile** settings and settings for connectors configured in the default **Workspace** are held in subfolders of the Application Directory. Each connector within the default **Workspace** has a dedicated subfolder within the `data` directory, named the same as the ConnectorID. Each connector's **Send** and **Receive** folder are held within this directory tree. Connectors configured in non-default **Workspaces** are held in the **Workspaces Directory** using a scheme similar to the default **Workspace**. | | **Application Log** | The Application Log stores application-level errors and calls to application resources. The Application Log can be viewed on the Status page of the application, and the full log can be exported using the 'Download Log File' button in the top right. More specific information for an error during a particular transaction may be found in the Transaction Log. | | **ArcScript** | ArcScript is an XML-based language that you can use to write custom processing logic in a dedicated Script Connector, in the **Events** of other connectors, and during a data mapping/transformation step in the XML Map Connector. | | **Connector** | Connectors are the building blocks for creating a logical **Flow** of data processing. Each connector performs a single operation on a data file: Send a file over the network to a remote system/host, receive or download a file over the network, transform the file locally (within the bus), or route the file to another connector within the Flow (each of these operations is referred to as a **Transaction**). Each connector has a **Transactions** tab where files are processed and sent from, or received and written to. | | **Embedded Web Server** | Arc can be hosted on any web server, but the application comes bundled with an embedded web server that can be used to immediately get up and running. Running the Arc executable or .jar file launches the embedded web server. When hosting the application on an external server (IIS, Tomcat, Jetty, etc), the embedded web server should not be used. | | **ESB** | An Enterprise Service Bus is a platform that allows many separate systems (databases, business applications, file exchange systems) to talk to each other. Arc provides a simple and modular solution for integrating these data processing tools. | | **Events** | Events are triggered when a connector processes a **Message** (e.g. Sending or Receiving a file). Events provide an opportunity to write **ArcScript** to implement custom logic when these triggers occur. | | **Flow** | A Flow is a connected series of connectors. Connections between connectors are displayed via blue flow arrows in the **Flows Designer**. Files are automatically passed from the **Transactions** tab of one connector to the **Transactions** tab of the next connector. This creates a logical chain of data processing. Multiple Flows can be configured in the same canvas, or they can be split into separate **Workspaces**. | | **Flows Designer** | The Flows page of the application displays the visual Flows Designer, a canvas where **Flows** are configured. Create **Flows** by dragging connectors from the toolbox on the left into the canvas. Configure connectors by clicking on them within the canvas, and connect connectors by dragging the blue dot on the right side of the connector onto the left side of another connector. | | **Input Mapping** | Database Connectors (and similar connectors like MySQL, SQL Server, CData) use Input Mappings to insert data into back-end systems. Each Input Mapping targets a single table/view (or virtual table/view) within the database/application. Input Mappings determine which columns within the table to insert data, and support advanced insert logic like UPSERTs (inserting or updating depending on a column value), lookup queries (fetching a value before inserting it in the table), and simultaneously inserting into multiple tables that are related via foreign key relationships. Input Mappings are modeled as XML, so data should be formatted as XML before being processed/sent by database-like connectors. | | **Installation Directory** | The folder that contains Arc executables. For the Windows edition of Arc, this directory also contains `www` and `www_services` folders. The Java edition includes the `Uninstaller` and `webapp` folders. | | **Mapping** | There are several kinds of mappings in Arc: **Input Mappings**, **Output Mappings**, and XML Mappings. Typically, 'mapping' refers to XML Mappings, which are how Arc handles data transformation within the bus. Many connectors can translate files into XML and vice versa, and the powerful XML Map Connector transforms one XML structure into another. A combination of these connectors allows for transforming between arbitrary data formats using XML as an intermediary format. | | **Message** | Arc processes raw data files, and also maintains metadata (headers) containing information about how the file has been processed within the application. The combination of the raw data file (payload) and metadata headers is called a message. Message headers can be viewed by clicking on the filename of any transaction entry in the Transaction Log or the Transactions tab in a specific connector. | | **MFT** | MFT is an acronym for Managed File Transfer, and refers to any standard protocol to securely send and receive data to and from other parties. Common MFT protocols are AS2, AS4, OFTP, FTP, and SFTP. Arc includes many MFT connectors that often serve as the entry point or exit point for data in a **Flow**. | | **Output Mapping** | Database Connectors (and similar connectors like MySQL, SQL Server, CData) use Output Mappings to pull data from back-end systems. Each Output Mapping targets a single table/view (or virtual table/view) within the database/application. Output Mappings determine which columns within the table should be returned, and support WHERE clauses (rules), pulling data from multiple tables, and the ability to only fetch new or recently modified records. Output Mappings are modeled as XML, so data returned from a database-like connector are formatted as an XML file. | | **NodeID** | The NodeID is a one-way hash of the machine name (or hostname in Java) that is used for licensing purposes. Each standard (non-cloud) license is tied to a single NodeID. The NodeID value is included in a license file (`arc.lic`) that the application checks to verify licensing status. | | **Profile** | The Profile page allows for configuring global settings that are used across the application. Some file transfer protocols like AS2, AS4, and OFTP require configuring a local profile to send and receive files. The Profile also includes security settings, a list of Users permitted to consume the **Admin API**, and global alerts for errors that occur in the application. | | **Receive** | Receiving is the action performed by connectors when generating an output file (**Message**). For networked connectors like AS2, FTP, and SFTP, Receiving means a file was received passively (e.g. AS2) or actively downloaded (e.g. SFTP) over the network. For local transformation connectors like X12, EDIFACT, CSV, and XML Map, Receiving means writing an output file after processing (translating/transforming) a local input file. | | **Receive Automation** | Some connectors can be scheduled to generate output files (e.g. poll a remote server for files to download or execute an output procedure at a specified interval). When enabled, a Receive Interval must be configured to determine when the connector **Receives**. Receive Intervals can be scheduled every X minutes, every X hours, daily at a specific time, weekly on a specific day, or via cron expression to maintain full granular control of the schedule. | | **Send** | Sending is the action performed by a connector when processing incoming files. For networked connectors like AS2, FTP, and SFTP, Sending means the file is sent or uploaded to a remote system or host. For local connectors like X12, EDIFACT, CSV, and XML Map, Sending means processing the file within the bus, transforming or translating it. | | **Send Automation** | Connectors support Send Automation settings to determine when and how input files will be processed/sent. When enabled, files that enter the connector's Transactions tab are automatically sent out or processed. Some networked connectors support Retry settings for retrying failed outgoing transmissions before raising an error in the application; this helps endure temporary network issues without causing a blocking failure. | | **Transaction** | A Transaction is any time a connector sends, receives, or processes a file. For local transformation connectors, a Transaction includes both a **Send** and a **Receive** (the connector Sends the original input file and Receives the output file once it has been transformed/translated). For networked connectors, a Transaction is typically a single Send (sending/uploading a file over the network) or a single Receive (receiving or downloading a file over the network). | | **Transaction Log** | The Transaction Log stores metadata about each **Transaction** processed by the application. The Transaction Log can be viewed on the Status page, and clicking on a filename within the Transaction Log brings up metadata about the **Message** that was processed and allows for downloading the raw data file or all logfiles generated during the **Transaction**. | | **View** | The **Flows Designer** allows for saving a particular position (a set of coordinates and zoom level) as a View when navigating the canvas. Clicking on a saved View returns the display to the same position that was shown when the View was saved. Saving Views helps navigate between different configured **Flows** when many connectors populate the canvas. | | **Workspace** | Workspaces allow for the separation of processes for organizational purposes. For example, the connectors required for an automated trading process with Walmart could be separated out into a new workspace that is separate from the connectors required for an automated trading process with Target. | | **Workspaces Directory** | A folder within the **Application Directory** that contains configuration files, raw data files, and logfiles for any non-default **Workspaces** in the application. | # Getting Started Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/index Overview of CData Arc's core features, including flows, connectors, mapping, scripting, and EDI trading partner management. CData Arc is a web application that provides a suite of Connectors for creating, executing, and monitoring custom data-integration flows. The Arc visual flow designer simplifies the process of integrating Managed File Transfer (MFT), EDI messaging, API management, data manipulation, and back-end integration. Arc workflows are modular, which means they can handle entire end-to-end integrations or perform specific data-management functions as part of a larger data ecosystem. See [Architecture](./architecture) for detailed information about the principles and design of Arc. ## Key Features * **Managed File Transfer:** Secure messaging capabilities including AS2, AS4, SFTP, FTPS, FTP, OFTP, RNIF, SOAP, and so on. * **Data Mapping:** Mapping and translation between data and file formats including EDI, EDIFACT, X12, IDoc (SAP), CSV, JSON, XML, and so on. * **REST-based APIs:** Accessible management through simple standards-based interfaces like SOAP, REST, HTML, RSS, ATOM, JSON, XLS, and CSV. * **Flexible Deployment Options:** A modern web-based application that is deployable anywhere - on-premises (Windows, Linux/UNIX, macOS) or in the cloud. * **Workflow Engine:** Easily build and automate workflows between applications such as Salesforce, MailChimp, Shopify, Google Apps, Dropbox, Microsoft Dynamics, NetSuite, and more. ## Quick-Start Guide See the [Quick-Start Guide](./quick-start-guide) for a set of links enabling you to quickly find the Arc information you need. ## Installation and Configuration See [Installation and Configuration](./installation/installation-configuration) for guides on Arc setup including:
  • Using the embedded web server for simple setup and deployment
  • Hosting the server in IIS or an external Java servlet container
  • Configuring alerts to notify users when an error or notable event occurs
  • Clustering multiple instances of Arc for availability and/or performance
  • Server logging
## Flows Flows are the heart of the application and offer a flexible way to build scalable workflows to move and manipulate data. See the [Flows](../flows) section for more information about creating flows in Arc. ## EDI Trading Partner Management Console The Trading Partner Management Console (TPMC) is a central place where users can go and configure an EDI partner relationship, map any necessary documents, and have that partner's details as well as the document mapping be reusable. See [EDI Trading Partner Management Console](../tpmc) for details on how to use the TPMC. ## Connectors Flows consist of a set of Connectors. Each connector performs one of the following roles in the Arc flow: * Connect to remote servers, trading partners, databases, and applications * Transform files into different formats or manipulate the data within a file * Route files through the flow to provide complex branching logical processes which provide connectivity to applications, databases, servers, as well as data transformation See the [Connectors](../connectors) section to learn more about specific connectors in Arc. ## Mapping Both the [TPMC](../tpmc) and [XML Map connector](../connectors/xml-map/xml-map) rely on mapping to define how data is translated between various XML data structures. Mapping plays an important role in facilitating data transformation and exchange in Arc. See [Mapping](../mapping) to learn how to define mappings. ## Admin API The Admin API is a REST interface for managing Arc programmatically—you can create and configure connectors and workspaces, manage certificates and files, read logs and transaction metadata, and run administrative actions without using the browser-based UI. See the [Admin API](../admin-api) section for authentication, resources, and actions. For a complete, per-operation listing of every endpoint, see the [API Reference overview](../api-reference/overview). ## How-To Guides The How-To Guides provide step-by-step walkthroughs for common Arc tasks, including configuring AS2, AS4, and OFTP connectors, building EDI flows, and backing up or migrating an Arc instance. See the [How-To Guides](../how-to) section for the full set of tutorials. ## Scripting Arc core functionality can be extended with ArcScript and Python, two scripting features included with the application. Event-based scripting is available within each connector, and a dedicated Script connector supports fully-customizable processing. See the [Scripting](../scripting) section to learn more about extending Arc functionality with ArcScript and Python. # arc.properties Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/arc-properties-configuration Reference guide for all supported configuration options in the CData Arc arc.properties file for the embedded Jetty web server. This reference guide covers all supported configuration options in CData Arc's `arc.properties` file. This file is used in the [embedded Jetty web server](./cross-platform-edition#configuring-the-embedded-jetty-server). To add any of these options to your Arc application, use a text editor to open the `arc.properties` file (located in the installation directory), then add the appropriate syntax. You can find examples of each option below. Arc reads `arc.properties` at startup. Restart Arc after you edit the file for your changes to take effect. > [!TIP] > > The following options are available in any Arc`.properties` files you have in your implementation. ## Using Variables You can provide variables in the configuration options, but you must keep these considerations in mind: * You can use variables for property *values*, but you cannot use them in property *names*. * Variable names are case-sensitive. * System properties take precedence over environment variables. If a variable is not found in the system property or the environment variable, it defaults to an empty string. * Variables cannot be nested. For instance, `cdata.warPath=${cdata.prefix${cdata.arc}}/webapp` is not allowed. The following example uses variables in configuration options to perform the following: * Define the initialization parameters * Set custom headers ``` cdata.initParameters=ARCCLOUD_SERVICE_DEFINITION_PATH:${cdata.home}/service.json,APP_LOGS:${cdata.logdb} cdata.http.customHeaders=Product-Version:[${ARC_ISSUER} 2025];/api.rsc,Product-Issuer:${ARC_ISSUER} ``` ## Configuration Options ### cdata.app.db Defines the connection string to the backend database that is used by Arc. `cdata.app.db=jdbc:cdata:mysql:server=localhost;port=3306;database=mysql;user=MyUserName;password=MyPassword` ### cdata.app.directory Defines the path that Arc should use as its application directory. `cdata.app.directory=/mnt/shared/arc` ### cdata.app.messageDirectory Defines a separate directory for message data (the files and payloads that flow through connectors). When set, Arc writes message files to this directory instead of the application directory. Configuration data (such as connector settings, profiles, and certificates) always remains in the application directory. See [Configure the Message Directory](./cross-platform-edition#configure-the-message-directory) for full details. `cdata.app.messageDirectory=/mnt/message-data/arc` ### cdata.contextPath Defines the base URL path at which a web application is deployed. It acts as a prefix for all the URLs handled by the web application. `cdata.contextPath=/myapp` For example, if the application is hosted at `www.mydomain.com`, this ensures the receiving endpoint is `http://www.mydomain.com/myapp/pub/Receive.rsb`. ### cdata.encryption.key Specifies the path to an AES key file that contains a random character string, or a PFX (PKCS#12) certificate file, to use as the master key. The master key encrypts the encryption key that encrypts sensitive data such as passwords and secrets in Arc. Only required if you want to use your own key or store the key in a non-default location. `cdata.encryption.key=/var/opt/mykeys/mysecret.key` Alternatively, if you are using an AES key, you can use the `APP_KEY` environment variable to supply the file path to the key file. ### cdata.encryption.key.password Defines the password for a PFX file that is being used for the master key (set using [cdata.encryption.key](#cdata-encryption-key)). `cdata.encryption.key.password=MyS3curePa55w0rd` Alternatively, you can use an environment variable (for example, `ENCRYPTION_KEY_PASSWORD`) to store the password, and reference it here. `cdata.encryption.key.password=${ENCRYPTION_KEY_PASSWORD}` ### cdata.fips.enabled Boolean option that controls whether Federal Information Processing Standards (FIPS) mode is enabled. FIPS is a set of security standards developed by the United States federal government for use in computer systems to ensure security, interoperability, and data integrity. `cdata.fips.enabled=true` ### cdata.http.customHeaders Lets you provide custom headers as concatenated string key-value pairs in the following format: `key:value`. When variables are included, the string is split at the first occurrence of a variable, and the variable is replaced. Only values can use variables. `cdata.http.customHeaders=Product-Version:[${ARC_ISSUER} 2025];/api.rsc,Product-Issuer:${ARC_ISSUER}` ### cdata.http.gzipEnabled Boolean option that controls whether GZIP compression is enabled for HTTP responses. When enabled, Jetty automatically compresses responses to reduce bandwidth usage, which improves performance for clients that support GZIP. `cdata.http.gzipEnabled=true` ### cdata.http.host Sets the host/IP address that Jetty's HTTP server should bind to. This determines which network interfaces the server listens on for incoming HTTP requests. `cdata.http.host=192.168.0.27` ### cdata.http.idleTimeout Controls the idle timeout for HTTP and TLS connections. This setting determines how long Jetty waits (in milliseconds) for activity on a connection before closing it. `cdata.http.idleTimeout=60000` ### cdata.http.maxFormSize Defines the maximum allowed size (in bytes) for HTTP form submissions (such as application/x-www-form-urlencoded and multipart/form-data requests). It limits the amount of data that can be processed in form submissions, including file uploads via forms. `cdata.http.maxFormSize=2000000` ### cdata.http.port Sets the HTTP port that Arc runs and listens on. `cdata.http.port=4080` ### cdata.http.proxyMode Boolean option that determines whether Jetty is running behind a reverse proxy or load balancer. When enabled, Jetty properly handles proxied requests, including headers like X-Forwarded-For and X-Forwarded-Proto. * When `true`, Jetty trusts X-Forwarded-\* headers from the proxy. * When `false`, Jetty ignores those headers and assumes direct client connections. This ensures proper client IP detection, HTTPS redirection, and host resolution when running behind a proxy. `cdata.http.proxyMode=true` ### cdata.http.requestHeaderSize Sets the maximum size (in bytes) of the HTTP request line and all request headers combined. The default is `8192`. If a request exceeds this limit, Jetty rejects it with a `414 URI Too Long` or `431 Request Header Fields Too Large` error. Increase this value if your clients send long URLs (for example, OData requests with large `$filter` expressions) or large headers. Because the request line and headers share the same limit, a long URL leaves less room for headers. The value must be a whole number. `cdata.http.requestHeaderSize=32768` ### cdata.initParameters Defines initialization parameters for the servlet. These parameters are concatenated string key-value pairs that are accessed by the servlet or filter at runtime. They must be set in a comma-delimited string in the following format: `key:value`. When variables are included, the string is split at the first occurrence of a variable, and the variable is replaced. Only values can use variables. `cdata.initParameters=ARCCLOUD_SERVICE_DEFINITION_PATH:${cdata.home}/service.json,APP_LOGS:${cdata.logdb}` ### cdata.loginService.ldap This is interpreted as `cdata.loginService.ldap.*`. These properties are used to configure Lightweight Directory Access Protocol (LDAP) authentication in Jetty, which allows users to authenticate against an LDAP server. A number of properties are supported. At a minimum, the following are required when you use LDAP with the embedded Jetty server: ``` cdata.loginService.ldap.hostname cdata.loginService.ldap.bindDn cdata.loginService.ldap.bindPassword ``` A complete LDAP section of `arc.properties` might look like the following: ``` ;; LDAP cdata.loginService.ldap.enabled=true cdata.loginService.ldap.bindDn="CN=ldapuser,DC=example,DC=org" cdata.loginService.ldap.hostname="127.0.0.1" cdata.loginService.ldap.bindPassword="Adminpassword1" cdata.loginService.ldap.port="1389" cdata.loginService.ldap.debug=true cdata.loginService.ldap.authenticationMethod="simple" cdata.loginService.ldap.userBaseDn="DC=example,DC=org" cdata.loginService.ldap.userRdnAttribute="cn" cdata.loginService.ldap.userPasswordAttribute="userPassword" cdata.loginService.ldap.forceBindingLogin="true" cdata.loginService.ldap.roleBaseDn="DC=example,DC=org" cdata.loginService.ldap.roleNameAttribute="cn" cdata.loginService.ldap.roleMemberAttribute="member" cdata.loginService.ldap.useLdaps="false" ``` See [Configuring LDAP Authentication](./cross-platform-edition#configuring-ldap-authentication) for additional context, and the full list of supported properties. ### cdata.loginService.ldap.enabled Boolean that determines whether LDAP authentication is enabled in Jetty. This property is typically used to enable or disable the entire LDAP-based login service, based on your configuration needs. `cdata.loginService.ldap.enabled=true` ### cdata.requestLog.enabled Boolean that controls whether Jetty logs HTTP requests to an access log. When enabled, Jetty records details of incoming requests, such as client IP, request method, response status, and response time. `cdata.requestLog.enabled=true` ### cdata.requestLog.filenameDateFormat Specifies the date format pattern Jetty appends to log filenames when rotating logs. This helps organize logs by day, hour, or custom intervals. * Used with `cdata.requestLog.filename` to generate rotating log filenames. * The format follows Java's `SimpleDateFormat` pattern. The default is `yyyy_MM_dd`, which creates a new log file daily. * If empty (`""`), Jetty disables date-based rotation. `cdata.requestLog.filenameDateFormat=MM_dd_yyyy` ### cdata.requestLog.retainDays Specifies how many days Jetty keeps old access logs before automatically deleting them. * The default is `31`, which means Jetty keeps logs for 31 days before deleting them. * When set to `0`, Jetty disables log retention, which means logs are never deleted automatically. * Used with `cdata.requestLog.filenameDateFormat` to manage rotating log files. `cdata.requestLog.retainDays=90` ### cdata.session.timeout Defines the maximum duration of inactivity (in seconds) before a user's session expires in a Jetty web server. * The default is `600` seconds (10 mins). * Values less than or equal to `0` are ignored. `cdata.session.timeout=1200` ### cdata.tempDirectory Defines the location of the temp directory used by the application. `cdata.tempDirectory=/tmp/arc` ### cdata.tls.keyStoreType Specifies the type of keystore Jetty should use for TLS/SSL certificates. A keystore contains the private key and SSL certificate necessary for Jetty to establish secure HTTPS connections. Common types include JKS, PKCS12, BKS and PEM. `cdata.tls.keyStoreType=PEM` ### cdata.tls.keyStorePassword Defines the password for the specified keystore. `cdata.tls.keyStorePassword=Password123` ### cdata.tls.keyStorePath Specifies the file path to the keystore that Jetty should use for TLS/SSL encryption. `cdata.tls.keyStorePath=/pathto/mykeystore.jks` ### cdata.tls.port Specifies the port number that Jetty uses for TLS (HTTPS) connections. This is the secure alternative to `cdata.http.port`, which is used for plain HTTP traffic. `cdata.tls.port=443` ### cdata.tls.sniHostCheck Boolean that controls whether Jetty enforces hostname verification during Server Name Indication (SNI) processing in TLS/SSL connections. `cdata.tls.sniHostCheck=true` ### cdata.warPath Sets the path of the `arc.war` file, if it is located elsewhere. `cdata.warPath=${cdata.home}/webapp/arc.war` # Cloud Marketplaces Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/cloud-marketplaces Login credentials for CData Arc hosted on Microsoft Azure and Amazon Web Services (AWS) cloud marketplaces. CData Arc is available through the Microsoft Azure and Amazon Web Services (AWS) cloud marketplaces. This page details login information for these cloud-hosted services. ## Microsoft Azure ### Login Credentials After configuring Azure and launching your session, use these credentials to log into Arc: **Username**: `admin`\ **Password**: `Arc26:` followed by the machine name. For example, `Arc26:1a2b3c4d5e6f` ## Amazon Web Services (AWS) ### Login Credentials After configuring AWS and launching your session, use these credentials to log into Arc: **Username**: `admin`\ **Password**: `Arc26:` followed by the instance ID. For example, `Arc26:1a2b3c4d5e6f` # Cross-Platform Edition Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/cross-platform-edition How to install, configure, and run CData Arc's Cross-Platform edition with embedded Jetty, Tomcat, WebSphere Liberty, or external Jetty. The Cross-Platform edition of CData Arc includes an embedded Eclipse Jetty web server so that you can run Arc without any external server configuration. In addition, the installation includes a WAR file that can be deployed to any Java servlet container like Apache Tomcat, Eclipse Jetty, or IBM WebSphere Liberty. Arc requires that Java 17 or newer is installed on the machine. Deploying to an external Java servlet container requires Servlet 3.0 (Jetty 8+, Tomcat 7+, WebSphere Liberty 8.5+, and so on). > [!NOTE] > > No matter how you choose to deploy Arc, the files and folders in `AppDirectory` must be readable and writable by the user who runs the application. The service installer that is included with the Java Edition setup uses **cdatarc** as this user. > > If the application was run previously as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, you should use a command similar to this example: > > `sudo chown -R cdatarc:cdatarc /opt/arc` ## Windows In Windows, Arc is installed as a service by default. To access the application, you must first ensure that the service is running. Once the service is running, you can access the Arc admin console by opening a web browser and entering [http://localhost:8080/](http://localhost:8080/) in the URL field. You can also run the application without the service through a `java` command. CData recommends using the service, but this method can be useful for certain configurations. ### Starting and Stopping the Service You can start and stop the service in any of the following ways: * [Start menu shortcuts](#start-menu-shortcuts) (recommended) * The [Services Management Console](#services-management-console) * [Command prompt](#command-prompt) commands #### Start Menu Shortcuts The Arc installer creates start menu shortcuts that allow you to easily use the application. To access these shortcuts, open the **Start** menu and expand the CData Arc folder. These shortcuts are available: * **Launch CData Arc Admin Console**: Opens a web browser window in your default web browser to the admin console URL, [http://localhost:8080/](http://localhost:8080/). If the service is not running, the web browser returns an error. * **Start CData Arc**: Starts the Arc service. By default, this service runs when Windows starts so you do not need to run this command every time you run the application. * **Stop CData Arc**: Stops the Arc service. This action is necessary when you [upgrade Arc](./how-to-upgrade). #### Services Management Console To open the Services Management Console, open the start menu and type *services*. Select the **Services** application that appears. Scroll down to the service called **CData Arc**. If that service is running, the **Status** column says **Running**. Right-click the service to access options to **Start**, **Stop**, and **Restart** it. #### Command Prompt Advanced users can use a Windows command prompt to issue commands manually to the service. Open a command prompt and change the directory to the installation folder (by default, `C:\Program Files\CData\CData Arc`). > [!NOTE] > > You can also use a Microsoft PowerShell window to issue these commands, but the syntax is slightly different. Modify the commands accordingly if you use PowerShell. To start the Arc service, issue this command: ```sh theme={null} arc.exe -start ``` Use the following commands to restart or stop the service: ```sh theme={null} arc.exe --restart --ServiceName "CData Arc" arc.exe --stop --ServiceName "CData Arc" ``` ### Launching Without the Service To run Arc without starting the service, open a command prompt in the installation folder. Issue the following command to start the application: ```sh theme={null} java -jar arc.jar ``` ## Linux After you [install Arc to a location of your choice](./installation-configuration#cross-platform-edition), you can run Arc as a service or run the application manually. CData recommends using a service if you use Arc for critical applications. ### Running Arc as a Service Running Arc as a service enables the application to run independently from any user process and to restart automatically upon rebooting. This is the preferred method for critical applications. CData recommends creating a daemon to manage the Arc application on Linux. A script that is included in the installation package can do this automatically. You can also create the daemon manually if necessary. The safest way to configure the Arc daemon is to run the script that is included in the installation package, provided that the system uses the *systemd* daemon manager: ```bash theme={null} bash ./service.sh ``` This script creates a daemon called `arc.service`. You can then use `systemctl` to manage the daemon: ```bash theme={null} systemctl enable arc systemctl start arc ``` ### Running the Standalone Application To start Arc without creating a service, use the terminal to open the arc.jar file in the installation directory with a configuration file argument, as shown here: ```sh theme={null} java -jar arc.jar ``` ## Configuring the Embedded Jetty Server Arc comes pre-configured to work immediately in any environment. However, you can customize the way you access the data that is exposed in Arc by [generating the arc.properties file](#generating-the-arc-properties-file) in the Arc installation directory (for Windows, this is `C:\Program Files\CData\CData Arc` by default). For details on all of the `arc.properties` configuration options, see [arc.properties Configuration](./arc-properties-configuration). > [!TIP] > > These configuration options are available in all `.properties` files in your implementation. ### Generating the arc.properties File Before you can make any customizations to the embedded Jetty server, you must create the `arc.properties` file. Run the following command in the installation directory where arc.jar is located: ``` java -jar arc.jar -GenerateProperties ``` This creates the `arc.properties` file in the installation directory. The file contains parameters that you can modify to [change the port](#changing-the-port) or to [enable TLS/SSL](#enabling-tls/ssl). > [!NOTE] > > Once you have generated this file, upgrades to Arc do not overwrite it. #### Changing the Port To change the port on which the embedded server listens: 1. Locate the `arc.properties` file in *`InstallationDirectory`* and open it in a text editor. 2. Locate the following line where the port is set: `cdata.http.port=8080` 3. Change this value to the port number that you want. #### Enabling TLS/SSL To enable TLS/SSL connections (HTTPS), you also need to modify the `arc.properties` file in *`InstallationDirectory`*, as follows: 1. Set the `cdata.tls.keyStoreType` setting to the type of keystore to use. Valid values include **jks**, **pkcs12**, and **jceks**. 2. Set the `cdata.tls.keyStorePath` setting to the path of the keystore to use. Note that `${cdata.home}` might be used to refer to *`InstallationDirectory`*. 3. Set the `cdata.tls.keyStorePassword` setting to the password for the keystore. 4. Set the `cdata.tls.port` setting to the port that should be used to host the server. 5. (Optional) Set the `cdata.http.port` setting to an empty string to disable plaintext connections. The final result should look like this: ``` ;; HTTP cdata.http.port= ;; TLS cdata.tls.port=8443 cdata.tls.keyStoreType=PKCS12 cdata.tls.keyStorePath=${cdata.home}/mycertificate.pfx cdata.tls.keyStorePassword=mypassword ``` > [!NOTE] > > If you obtain an external private key for configuration in Arc, be sure to change the owner of the certificate to the service account that is used to host Arc (**cdatarc:cdatarc**). #### Pointing Arc to a Different .properties File Use the `-config` parameter to point Arc to a configuration file other than the default `arc.properties` file. For example, if you run the following command, Arc looks for a `test.properties` file: ``` java -jar arc.jar -config test ``` ### Generating a Jetty XML File In most deployments, the `arc.properties` file provides the full range of necessary configuration options for the embedded Jetty server. However, if you require a more complex deployment, you can generate a Jetty XML file that serves as a starting point for further modifications. To generate this file, run the following command in the Arc installation directory where arc.jar is located: ``` java -jar arc.jar -GenerateXML ``` When the command completes, the arc.xml configuration file appears in the `webapp` folder. As long as it remains in that folder, it is used to start the application. ### Starting and Stopping the Server If you use the `service.sh` script to set up an Arc service, see [Running as a Service](#running-as-a-service). Otherwise, the [Running In-Process](#running-in-process) section is applicable. #### Running In-Process Start the embedded Jetty server by executing the arc.jar file that is extracted from the application download during setup. You can use standard Java syntax, as shown below, to execute this file and start the server: ```bash theme={null} java -jar arc.jar ``` To stop the server, pass the `-stop` parameter to this command: ```bash theme={null} java -jar arc.jar -stop ``` #### Running as a Service You can manipulate the Arc service by using standard system service commands, referencing **arc** as the name of the service. To start the service, submit this command: ```bash theme={null} systemctl start arc ``` To stop the service, submit this command: ```bash theme={null} systemctl stop arc ``` To restart the service, submit this command: ```bash theme={null} systemctl restart arc ``` ### Configuring LDAP Authentication The following steps configure Arc to use LDAP to authenticate users if you use the [embedded Jetty web server](#configuring-the-embedded-jetty-server). #### Configure arc.properties If it doesn't already exist, [generate the default arc.properties](#generating-the-arc-properties-file) file with the following command: `java -jar arc.jar -GenerateProperties` The following mandatory setting instructs the embedded Jetty server to use LDAP for authentication: `cdata.loginService.ldap.enabled=true` You might want to section this off inside the file in the same way as other sections: ``` ;; LDAP cdata.loginService.ldap.enabled=true ``` Insert any additional settings that are required by your specific LDAP server, along with the minimum settings required by Arc. At a minimum, Arc requires the following: ``` cdata.loginService.ldap.hostname cdata.loginService.ldap.bindDn cdata.loginService.ldap.bindPassword ``` The full list of available settings is below: ``` cdata.loginService.ldap.userIdAttribute cdata.loginService.ldap.debug cdata.loginService.ldap.forceBindingLogin cdata.loginService.ldap.bindPassword cdata.loginService.ldap.roleMemberAttribute cdata.loginService.ldap.useLdaps cdata.loginService.ldap.roleBaseDn cdata.loginService.ldap.bindDn cdata.loginService.ldap.userPasswordAttribute cdata.loginService.ldap.hostname cdata.loginService.ldap.userRdnAttribute cdata.loginService.ldap.roleObjectClass cdata.loginService.ldap.port cdata.loginService.ldap.authenticationMethod cdata.loginService.ldap.userBaseDn cdata.loginService.ldap.contextFactory cdata.loginService.ldap.userObjectClass cdata.loginService.ldap.roleNameAttribute ``` Following is an example of what a complete LDAP section of `arc.properties` might look like: ``` ;; LDAP cdata.loginService.ldap.enabled=true cdata.loginService.ldap.bindDn="CN=ldapuser,DC=example,DC=org" cdata.loginService.ldap.hostname="127.0.0.1" cdata.loginService.ldap.bindPassword="Adminpassword1" cdata.loginService.ldap.port="1389" cdata.loginService.ldap.debug=true cdata.loginService.ldap.authenticationMethod="simple" cdata.loginService.ldap.userBaseDn="DC=example,DC=org" cdata.loginService.ldap.userRdnAttribute="cn" cdata.loginService.ldap.userPasswordAttribute="userPassword" cdata.loginService.ldap.forceBindingLogin="true" cdata.loginService.ldap.roleBaseDn="DC=example,DC=org" cdata.loginService.ldap.roleNameAttribute="cn" cdata.loginService.ldap.roleMemberAttribute="member" cdata.loginService.ldap.useLdaps="false" ``` #### Create LDAP Users in Arc In order for users to log into Arc via your LDAP server, you must add each LDAP user to Arc. This allows Arc to cross-reference the user trying to log in with the configured LDAP server. Follow these steps to create each user (see [User Management and Roles](../administration/settings/user-roles) for more details on managing users): * Start Arc and log in as a Arc admin user. * Create all LDAP users who need access to Arc. Click the gear icon in the navbar and choose **Users**. The Arc users must be identical to the LDAP users. For example, if your LDAP users are `user01` and `user02`, you must use the same usernames in Arc. Once you have added all users, you can stop the application. #### Test the Configuration Once you have created the `arc.properties` file with the necessary LDAP settings, added your LDAP users to Arc, and confirmed that their configuration is correct based on your LDAP server's requirements, you are ready to test the functionality. Start Arc using `java -jar arc.jar` or by starting the Arc service. When you are presented with the login screen, try to log in using one of your LDAP users. Enter the username and password as it exists on the LDAP server (which must be identical to how you entered it in the **Users** section of Arc). The login process checks the logged-in user against the users configured in Arc and then against the LDAP server to ensure that the user is present and allowed to access the application. If the configuration was successful, you will be logged into the application as that user. #### Troubleshooting LDAP Configurations If you have trouble authenticating to your LDAP server, CData recommends that you test the LDAP connection and filter with [Apache Directory Studio](https://directory.apache.org/studio/). You can apply the same connection settings and search filter to the `arc.properties` or `login.config` files. ##### Creating a Debug Log for Failed Authentication Attempts You can enable debugging in the `arc.properties` file with this setting: `cdata.loginService.ldap.debug`. You can also direct additional logging information to the console output and webserver logs. Create a file called `arc.logging.properties` in the same location as the `arc.properties` file and the `webapps` folder in your application installation directory with the following contents: ``` handlers = java.util.logging.ConsoleHandler, arc.ArcRolloverFileHandler java.util.logging.ConsoleHandler.level = FINE arc.ArcRolloverFileHandler.filename = yyyy_mm_dd.log arc.ArcRolloverFileHandler.formatter = java.util.logging.SimpleFormatter arc.ArcRolloverFileHandler.level = FINE # Enable LDAP and Jetty JAAS module debug logs org.eclipse.jetty.jaas.level = FINE com.sun.jndi.ldap.level = FINE ``` After you restart the service, additional logging regarding the LDAP authentication attempts is output to the console. ## Configuration in Tomcat ### Deploy the WAR File You have two options for deploying a WAR file to Tomcat. * Copy the WAR file into the `webapps` folder. * Deploy the WAR file from within the management console in Tomcat. The Apache Tomcat documentation covers this method in more detail. See the documentation for your version of Tomcat. It is possible that the WAR file might exceed the default maximum size allowed for file uploads in Tomcat. To overcome errors during deployment, you can edit the `web.xml` file of the manager application to allow larger files. Depending on your Tomcat configuration, this file might reside in `/usr/share/tomcat7-admin/manager/WEB-INF` or in a similar directory. In this file, you can change the size, in bytes, of the maximum allowed file size. For example, to allow deployment of a 200MB WAR file, edit the following values to change the maximum allowed file size: ```xml theme={null} 209715200 209715200 0 ``` ### Configure the Java Authentication and Service (JAAS) To enable Arc to manage users dynamically within the application, you must configure the JAAS as described in the following guide. This guide uses `arc.xml` to control the configuration of Tomcat and define any context overrides. CData recommends that you use `arc.xml` instead of `server.xml` to control the Tomcat configuration. We also recommend that you relocate any application context overrides, such as `APP_DIRECTORY` or `APP_DB`, to `arc.xml` where the JAASRealm module is defined. > [!NOTE] > > This is a required first step for any Tomcat setup with Arc. #### Create the Login Module Create a JAAS configuration file with the name `jaas.config` in this folder: `$CATALINA_BASE/conf/`. Include the following content in `jaas.config` to use standard authentication: ```xml theme={null} Arc { arc.LoginModule required; }; ``` > [!NOTE] > > If you want to use LDAP, continue with the standard login module setup. Once that's complete, follow the steps in [Using LDAP to Authenticate Users](#use-ldap-to-authenticate-users). #### Create (or Modify) the JAASRealm Module 1. Check to see whether you have an `arc.xml` file in `$CATALINA_BASE/conf/Catalina/localhost/`. If you do, edit `arc.xml` and add the XML context block below. If `arc.xml` does not exist at that path, you need to create it, then add the XML context block below. This should be the only content in `arc.xml`. ```xml theme={null} ``` > [!NOTE] > > Depending on how your Tomcat instance is configured, this path might be slightly different. In this example, `Catalina` refers to the engine name, and `localhost` is the host name defined in `server.xml`. 2. Update the `` element in your `server.xml` configuration file for the Tomcat server by setting the `copyXML` attribute to true, as shown below: ```xml theme={null} ... ``` > [!NOTE] > > If an application-specific context is present in `server.xml`, it takes precedence over `arc.xml`. CData recommends using `arc.xml` over `server.xml` for any context overrides. For example: > > ```xml theme={null} > > ... > > ``` #### Make the Login Module Visible The Java virtual machine (JVM) must be directed to the login module (`jaas.config`) for the configuration to be visible. Set the `java.security.auth.login.config` system property on the JVM to the path of the `jaas.config` file by appending the following line to the `$CATALINA_BASE/conf/catalina.properties` file: ```bash theme={null} java.security.auth.login.config=${catalina.base}/conf/jaas.config ``` #### Use LDAP to Authenticate Users Before you can configure Arc to use LDAP with Tomcat, follow the [Configure the Java Authentication and Service (JAAS)](#configure-the-java-authentication-and-service-jaas) instructions so that you can create your admin user, which is required before you can add LDAP users to Arc. Once you have configured the login module and can successfully log into Arc as the admin user, follow these steps to configure LDAP support. 1. Log in to Arc as the admin user. To create all LDAP users who need access to Arc, click the **Settings** gear icon on the navbar and choose **Users**. The Arc users must be identical to the LDAP users. For example, if your LDAP users are `user01` and `user02`, you must use the same usernames in Arc. 2. Modify the `$CATALINA_BASE/conf/jaas.config` file that you created in [Create the Login Module](#create-the-login-module) to work with your LDAP server. You must add some configuration options to `jaas.config`. a. Ensure that the LDAP login module is required by adding `com.sun.security.auth.module.LdapLoginModule REQUIRED`. You also need to make the login module you created previously optional. To do this, set `arc.LoginModule optional;`. b. Add the required LDAP module configuration options. At a minimum, the following options are needed: `userProvider`, `authIdentity`, `userFilter` and `useSSL`. The values that you provide for these options are specific to your LDAP server and requirements: consult your server admin and/or your LDAP documentation to determine the values. Here is an example of what you can expect it to look like — notice that `arc.loginModule` is set to `optional` and `com.sun.security.auth.module.LdapLoginModule` is set to `REQUIRED`: ``` Arc { com.sun.security.auth.module.LdapLoginModule REQUIRED userProvider="ldap://openldap:1389/ou=users,dc=example,dc=org" authIdentity="CN={USERNAME},OU=users,DC=example,DC=org" userFilter="(cn={USERNAME})" useSSL=false; debug=false; arc.LoginModule optional; }; ``` > [!NOTE] > > If any value contains the special token `{USERNAME}`, that token is replaced with the supplied username value at login. Once you complete these steps, your LDAP users can log into Arc. ### Configure Data Directory Permissions Give the user of the process that runs the Java servlet container Read/Write access to the data directory in the appropriate location, as follows: * **Windows:** `C:\ProgramData\CData\Arc\` * **Linux:** `~/cdata/arc` Restart the Tomcat server for the changes to take effect. You can now log in to the application. ## Configuration in WebSphere Liberty To configure Arc in WebSphere Liberty, follow these steps. ### Create the Arc Application in WebSphere Liberty > [!NOTE] > > This guide describes a basic installation of Arc in Liberty. If you need more of a custom Liberty installation, consult with your internal team to understand what additional settings or options might need to be included. 1. Download Liberty from [Open Liberty](https://openliberty.io/start/) or [WebSphere Application Server Liberty 24.0.0.12](https://www.ibm.com/support/pages/240012-websphere-application-server-liberty-240012). Arc expects the Web Profile 8 package. 2. Run this command to create a server named `arc`: * Windows: `.\bin\server.bat create arc` * Linux: `./bin/server create arc` 3. Copy the arc.war file to the `./usr/servers/arc/apps` directory. 4. If necessary, change the HTTP and HTTPS ports in the `httpEndpoint` element in `./usr/servers/arc/server.xml` (a [sample server.xml file](#sample-server-xml-file) is below). 5. Make any additional configuration application updates by editing the server.xml file. 6. Run this command to start the server: * Windows: `.\bin\server.bat start arc` * Linux: `./bin/server start arc` ### Sample server.xml File The following is a sample server.xml file that contains settings that Arc needs to run in Liberty. It also contains the settings required for JAAS configuration. The following sections provide more detail on portions of this file. ```xml theme={null} webProfile-8.0 ``` ### Configure the Java Authentication and Service (JAAS) The following settings in server.xml configure the JAAS and enable Arc to manage users dynamically in the Liberty application server. If you copied the server.xml sample file above, these elements are already present. Otherwise, follow these steps: 1. Configure the Arc login module for JAAS authentication by adding the following elements to server.xml: ```xml theme={null} ``` 2. Configure user groups and role mappings in the `webApplication` element of server.xml. ```xml theme={null} ``` The `application-bnd` section maps Arc's security roles to user groups, controlling what permissions different users have in the application. Each `security-role` element defines one of Arc's permission levels and associates it with a corresponding user group. Make sure you have a `security-role` element for **cdata\_admin**, **cdata\_standard**, **cdata\_support**, and **cdata\_user**. 3. Enable query parameters without equal signs (=) by adding the `AllowQueryParamWithNoEqual` property to the `webContainer` element. This setting allows Arc to properly handle certain URL query parameters that don't include an equal sign. ```xml theme={null} ``` 4. Restart Liberty. Now Arc is accessible and ready to use with the Arc login module. This securely manages users at the application level. To configure LDAP, read the following section. ### Use LDAP to Authenticate Users Before you can configure Arc to use LDAP with Liberty, follow the [Configure the Java Authentication and Service (JAAS)](#configure-the-java-authentication-and-service-jaas-2) instructions to create your local admin user. That user is required before you can add LDAP users to Arc. Once you have configured the login module and can successfully log into Arc as the admin user, follow these steps to configure LDAP support. 1. Edit server.xml to add the LDAP repository to Liberty. 1. In the `featureManager` element, add a new feature: `ldapRegistry-3.0`. ```xml theme={null} webProfile-8.0 ldapRegistry-3.0 ``` 2. Add the details for your LDAP registry after the `webContainer` element. ```xml theme={null} ``` 3. Change the `arc.LoginModule` control flag to `OPTIONAL`. ```xml theme={null} ``` 2. Log in to Arc as the local admin user. To create your LDAP users in Arc, click the **Settings** gear icon in the navbar and choose **Users**. The Arc users must be identical to the users in your LDAP server. For example, if your LDAP users are `user01` and `user02`, you must use the same usernames in Arc. 3. Save your changes and restart Liberty to complete the process. Once successfully configured, users signing into Arc are authenticated via LDAP. ### Debug Settings The [sample server.xml](#sample-server-xml-file) file includes a comprehensive logging configuration that captures detailed trace information for security, authentication, web container operations, and Arc-specific components. This can be useful if you are having configuration issues with LDAP or other settings. ```xml theme={null} ``` The `traceSpecification` attribute is set to `all` for several key component groups including security bindings, authorization, JAAS authentication, web container operations, and all Arc and RSSBus modules. This configuration provides thorough diagnostic information useful for troubleshooting authentication issues, authorization problems, and application-specific errors. The logs are written to the `${server.config.dir}/logs` directory with separate files for messages (`messages.log`) and trace data (`trace.log`). For production environments, you might want to reduce the logging verbosity to improve performance and minimize log file growth. To do this, change specific trace specifications from `all` to `info` or remove components you do not need to monitor. For example, if you are not troubleshooting security issues, you could simplify the configuration to `traceSpecification="arc.*=all:rssbus.*=all"` to focus only on Arc-specific logging. Conversely, if you need even more detailed diagnostics during troubleshooting, you can temporarily set `traceSpecification="*=all"` to enable comprehensive logging across all Liberty components. Be aware this generates large log files quickly and should only be used for short-term debugging. ### Configure Data Directory Permissions Give the user of the process that runs the Java servlet container read and write access to the data directory: * **Windows:** `C:\ProgramData\CData\Arc\` * **Linux:** `~/cdata/arc` Restart the Liberty server for the changes to take effect. You can now log into the application. ## Configuration in Jetty Although Arc comes with an embedded Jetty web server, you can also use the application with an external Jetty setup. ### Deploy the WAR File and arc.xml Copy arc.war into the `webapps` folder of `${JETTY_BASE}`. Also place the arc.xml file in the same `${JETTY_BASE}` folder. If you do not have arc.xml you need to create it. At a minimum, for a standard configuration of Jetty your arc.xml file needs to contain the following content: ```xml theme={null} / /var/lib/jetty/webapps/arc.war ArcRealm Arc arc.GroupPrincipal ``` ### Configure the Java Authentication and Service (JAAS) To configure the JAAS and to enable Arc to manage application users, you must perform the steps described in the following sections. #### Add the JAAS Module Submit the following command to install the JAAS module: ```bash theme={null} java -jar start.jar --add-modules=deploy,jsp,http,jaas ``` #### Create the Arc Login Module Create a login configuration file called login.config and place it at this path: `{JETTY_BASE}/etc/login.conf`. Place the following content in the login.config file: ``` Arc { arc.LoginModule required debug=true; }; ``` #### Update the Security Handler The Security Handler configuration is in the arc.xml configuration file. If you created the arc.xml file using the content in [Deploy the WAR File and arc.xml](#deploy-the-war-file-and-arc-xml), you can skip this step because this change is already present in that content. Otherwise, modify the `securityHandler` block as follows: ```xml theme={null} ArcRealm Arc arc.GroupPrincipal ``` ### LDAP in Jetty In order to configure LDAP when running Arc in Jetty, you need to configure Arc to run in Jetty using the standard Arc login module. Once Arc is up and running in Jetty using the Arc login module, follow these steps to configure Jetty to use LDAP for user authentication. 1. Add LDAP Users to Arc. 1. Login as your Admin user to add LDAP users to Arc. To create your LDAP users in Arc, click the settings gear icon in the navbar and choose **Users**. The Arc users must be identical to the users in your LDAP server. For example, if your LDAP users are `user01` and `user02`, you must use the same usernames in Arc. 2. When you have finished, stop Arc. 2. Create the JAAS configuration for the Jetty LDAP login module in login.conf. 1. Open the login.conf file located at `${JETTY_BASE}/etc/login.conf` and add a JAAS configuration for the Jetty LDAP module. A number of configuration settings are available, but the configuration of your JAAS depends on your specific requirements and your LDAP server configuration. Find the list of available JAAS configuration settings in the [Jetty documentation](https://jetty.org/docs/jetty/10/operations-guide/jaas/index.html#ldaploginmodule), as well as below: ``` debug useLdaps contextFactory hostname port bindDn bindPassword authenticationMethod forceBindingLogin userBaseDn userRdnAttribute userIdAttribute userPasswordAttribute userObjectClass roleBaseDn roleNameAttribute roleMemberAttribute roleObjectClass ``` 2. Here is an example of a login.conf file that is configured with an LDAP server. Remember, the required settings and values are dependent on your specific requirements and your LDAP server configuration. To determine the settings and values required for your setup, consult with your LDAP administrator or the documentation for your LDAP server. ``` Arc { arc.LoginModule optional debug=true; org.eclipse.jetty.jaas.spi.LdapLoginModule optional debug="true" contextFactory="com.sun.jndi.ldap.LdapCtxFactory" hostname="openldap" port="1389" bindDn="CN=ldapuser,DC=example,DC=org" bindPassword="Adminpassword1" authenticationMethod="simple" forceBindingLogin="true" userBaseDn="OU=users,DC=example,DC=org" userRdnAttribute="cn" userPasswordAttribute="userPassword" roleBaseDn="DC=example,DC=org" roleNameAttribute="cn" roleMemberAttribute="member" useLdaps="false"; }; ``` Notice that the `arc.LoginModule` and the `org.eclipse.jetty.jaas.spi.LdapLoginModule` are both set to optional. This allows both login modules to be used when trying to authenticate a user. If one login module fails to authenticate a user, the login falls back to the second module. If both login modules fail to authenticate a user, the login fails altogether. 3. Once login.conf has been updated with the necessary LDAP configuration, restart Arc. If it is configured correctly, your LDAP users can now log into Arc. ### Configure Data Directory Permissions Give the user of the process that runs the Java servlet container Read/Write access to the data directory: * **Windows:** `C:\ProgramData\CData\Arc\` * **Linux:** `~/cdata/arc` Restart the Jetty server for the changes to take effect. You can now log into the application. ## User Management Upon initial startup, Arc prompts you for the creation of a user with username and password credentials. After the first user is created, you can add, delete, and manage users on the **Users** tab of the **Settings** page in the application. When you deploy Arc to an external Java servlet (that is, when you **are not** using the embedded server that is included with the application), JAAS configuration is required in order to allow Arc to manage users. The previous sections detail the process of JAAS configuration for each specific external servlet. ## Find and Configure the Application Directory The Arc *`ApplicationDirectory`* folder contains all the data that is used by the application: configuration data, application data, logging data, certificates, and so on. The default location of *`ApplicationDirectory`* depends on whether Arc is hosted via the embedded web server or via an external Java servlet container. For the embedded web server, *`ApplicationDirectory`* is the same as *`InstallationDirectory`*. By default, that location is the following: ```bash theme={null} /opt/arc ``` When Arc is hosted in an external Java servlet container, *`ApplicationDirectory`* is relative to the home directory of the user who is running the server: `~/arc` In this path, '\~' resolves to the home directory of the user who is running the server that hosts the application. You can configure the *`ApplicationDirectory`* folder, which is useful in a variety of scenarios: * clustering multiple instances of Arc * using a shared network drive for application data * embedding Arc within other systems that access the same folders Changing *`ApplicationDirectory`* moves the application's data files. However, it does not move other application resources like EXE files, JAR files, and so on. These resources are held in the *`InstallationDirectory`* folder, which might be the same as *`ApplicationDirectory`*, but the location of those resources does not change if *`ApplicationDirectory`* is changed. ### Embedded Java Server When you use the Cross-Platform Edition with the embedded Jetty server, by default the *`ApplicationDirectory`* is the *`InstallationDirectory`*. To modify this, [generate the arc.properties file](#generating-the-arc-properties-file). Open the file in a text editor, then set the `cdata.app.directory` setting to the path of the desired directory. The following example demonstrates what this might look like when you set the data directory to a shared folder on a mounted drive: ```bash theme={null} cdata.app.directory=/mnt/shared/arc ``` If Arc can locate the `cdata.app.directory` path and it has the appropriate permissions to read and write at that path, it creates the data folder in the specified directory. ### External Java Server When you use the Cross-Platform Edition with an external Java servlet (any server other than the Jetty server that is included with the application), the details of configuring the application data directory depend upon the specific servlet that is used. Using the syntax that is appropriate for the specific servlet, the `AppDirectory` environment variable must be set to the path of the directory that you want. If Arc can locate the `AppDirectory` path and it has the appropriate permissions to read and write at that path, it creates the data folder in the specified directory. ## Configure the Message Directory By default, Arc stores message data (the files and payloads that flow through connector Send, Receive, Sent, and Pending folders) under the application directory. You can configure a separate message directory to manage message file storage independently from configuration data. Common reasons to configure a separate message directory include: * Backing up configuration data independently of high-volume message files * Placing message storage on a dedicated volume with separate capacity or performance characteristics * Satisfying compliance or security requirements about where processed data may reside Configuration data (such as connector settings, profiles, and certificates) always remains in the application directory regardless of this setting. > [!NOTE] > > Changing the message directory on an existing installation does not move previously-written message files. Only new messages are stored under the new directory going forward; existing message data stays in its original location under the application directory. ### Embedded Java Server To configure a separate message directory when using the embedded Jetty server, [generate the arc.properties file](#generating-the-arc-properties-file) if you have not done so already. Open the file in a text editor, then set the `cdata.app.messageDirectory` property to the path of the desired directory: ```bash theme={null} cdata.app.messageDirectory=/mnt/message-data/arc ``` Arc creates the directory automatically if it does not exist at startup. The configured path must be one to which the application has read and write permissions. Restart Arc for the change to take effect. ### External Java Server When using an external Java servlet container, use the syntax appropriate for your servlet to set the `MessageDirectory` environment variable to the path of the desired directory. Arc creates the directory automatically if it does not exist at startup. Restart the Java servlet container for the change to take effect. ## Configure the Application Database The Arc application database stores several tables of application data, including the following: * Transaction Log: metadata for each transaction that is processed by the application * Application Log: application-level errors and events * Access Log: requests to the application's web endpoint * Audit Log: user-made changes to the Arc configuration By default, Arc uses an H2 database that resides in *`ApplicationDirectory`* as the application database. This database is recommended for up to 100,000 transactions. Once you hit that number, CData recommends moving to an external database. You can configure the application to use an enterprise database like SQL Server, PostgreSQL, or MySQL. Arc monitors the number of transactions in your database and alerts you when the transaction count exceeds the threshold for your database type. When you are alerted (by a banner at the top of each page in the application), you must either use the [Cleanup Settings](../administration/settings/cleanup-settings) on the **Settings > Advanced** page to reduce the number of transactions, or [move to a different database](../administration/settings/app-db-migration) with a higher transaction threshold. Use the following table to determine which database is right for your implementation. | Database | Transaction Threshold | | - | - | | H2 | 100,000 | | Apache Derby (legacy) | 100,000 | | MySQL | 500,000 | | PostgreSQL | None | | SQL Server | None | > [!NOTE] > > Apache Derby is the legacy embedded database used before H2. It is still supported, but it shares the same transaction threshold as H2 and is not recommended for new implementations. > > For security reasons, if you ever switch to a different application database, you must run the [integrityResetTampering](../../scripting/operations/op-integrity-reset-tampering) operation to reset the hash chain. ### Embedded Java Server When you use the Cross-Platform Edition with the embedded Jetty server, the default application database is an H2 database in the *`ApplicationDirectory`*. To modify this, [generate the arc.properties file](#generating-the-arc-properties-file). Open the file in a text editor, then set the `cdata.app.db` setting to a Java Database Connectivity (JDBC) connection string that contains the appropriate connection parameters for the database that you want. The following examples illustrate this setting for MySQL, PostgreSQL, and SQL Server: #### MySQL ``` cdata.app.db=jdbc:cdata:mysql:server=localhost;port=3306;database=mysql;user=MyUserName;password=MyPassword ``` #### PostgreSQL ``` cdata.app.db=jdbc:cdata:postgresql:server=localhost;port=5432;database=postgresql;user=MyUserName;password=MyPassword ``` #### SQL Server ``` cdata.app.db=jdbc:cdata:sql:server=localhost;database=sqlserver;user=MyUserName;password=MyPassword ``` If Arc can establish a connection successfully with the `cdata.app.db` connection string, it uses that database as the application database. > [!NOTE] > > To reduce the possibility of deadlocks when using SQL Server as your application database, CData recommends that you ensure that READ\_COMMITTED\_SNAPSHOT is enabled. ### Generating an Encrypted Database Connection String Arc provides the ability to generate an encrypted connection string for your application database connection. You can use this encrypted connection string to specify the application database without storing your login credentials in plain text in the Arc configuration files. To generate an encrypted connection string, issue the following command in the installation directory where arc.jar is located, substituting your connection information for the example string in quotation marks: ``` java -jar arc.jar -EncryptConnectionString "jdbc:cdata:mysql:Server=serverName;Port=1234;Database=databaseName;User=userName;Password=passwordSample" ``` After you issue the command, the command window prints the encrypted string. For example: ``` ENCRYPTEDA:5TWyjE0U4GBEVNkd51SqbMXaBwVGqj5uyMfTHt48y/F5l3sDPZynNnPMb/ppXzuFTzFPywA1/SSKUgLy9nAuvA7RAdZj7WsKI1xAgDMmEFNQ68/lHjFYZpS10sIaoUloyZVDkj4uLHTZ0SaMD5PPUA== ``` You can then use this encrypted string in place of the plain text value for `cdata.app.db` as [shown above](#configure-the-application-database). ### External Java Server When you use the Cross-Platform Edition with an external Java servlet (any server other than the Jetty server that is included with the application), the details for configuring the application database depend upon the specific servlet that is used. Using the syntax that is appropriate for the specific servlet, choose one of the following approaches when you configure the server: * Define a JNDI data source to include the connection properties for the target database. * Set the `APP_DB` environment variable to a JDBC connection string. If Arc can use the JNDI data source or the `APP_DB` connection string to connect to a database, it uses that database as the application database. ### Specifying the Default Character Set (MySQL only) For MySQL versions such as MySQL 8.0 and later, the default character set (charset) for the database and its tables is UTF8 (specifically `utf8mb4`). However, in versions of MySQL prior to 8.0, the default charset is typically Latin1, which can cause issues with data that includes characters outside the basic Latin alphabet. If you use an older version of MySQL you can avoid these problems by configuring the database to use `utf8mb4` instead. You have two ways to make this change: update the existing database directly, or use MySQL's export and import tools to migrate the data to a new UTF8-encoded database. **Update the Database Directly** 1. Backup the database: `mysqldump -u root -p --default-character-set=latin1 --databases [database name] > backup.sql` 2. Change the default charset to UTF8: `ALTER DATABASE [database name] CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` 3. Generate SQL statements to change the default charset to UTF8 for all database tables: ``` SELECT CONCAT('ALTER TABLE ', table_name, ' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS alter_sql FROM information_schema.TABLES WHERE table_schema = '[database name]' AND table_type = 'BASE TABLE'; ``` 4. Execute the SQL statements generated in the previous step. **Use the Export and Import Tools** 1. Export the Data Definition Language (DDL) and data: ``` mysqldump -u root -p --default-character-set=latin1 --no-create-info [database name] > data.sql mysqldump -u root -p --no-data [database name] > schemas.sql ``` 2. Change the default charset in schemas.sql from `CHARSET=latin1` to `CHARSET=utf8mb4`. 3. Create a new database with UTF8 as the default charset: `CREATE DATABASE [new database name] CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` 4. Import to the new database: ``` mysql -u root -p [new database name] < schema.sql mysql -u root -p [new database name] < data.sql ``` ## Login Lockouts Arc automatically locks out users who enter incorrect passwords too many times in order to prevent brute-force attacks. By default, a user who enters six incorrect passwords within five minutes is locked out for thirty minutes. You can modify the lockout settings by editing the XML configuration file that governs the web-server behavior. These three settings are relevant to lockouts: * **LockoutFailedAttempts**: the number of incorrect passwords that trigger a lockout. Set LockoutFailedAttempts to 0 to disable lockouts. * **LockoutMinutes**: the duration of the lockout. The default duration is thirty minutes. * **LockoutTimeCheckPeriod**: the period after which the number of failed attempts is reset to 0. The default period is five minutes. ### Embedded Jetty Server You can modify lockout settings by [generating the arc.properties file](#generating-the-arc-properties-file) and adding a comma-separated list of *name:value* pairs to `initParameters`, as shown below: ```bash theme={null} cdata.initParameters=LockoutFailedAttempts:0 ``` ### Tomcat The syntax for editing the lockout settings in the Tomcat arc.xml file is the following: ```xml theme={null} ``` ## Common Issues and Solutions This section lists common issues that you can encounter when you deploy Arc in a Java environment. The recommended solution for each issue is included. For additional help, contact CData Technical Support at **[arcsupport@cdata.com](mailto:arcsupport@cdata.com)**. ### Issue #### Arc Fails to Start, or It Starts Using an AppDirectory Other Than Expected This error might indicate that Arc does not have the permissions that are required to access *`ApplicationDirectory`*. (*`ApplicationDirectory`* is a folder that stores critical information regarding the configuration of jobs, connections, transformations, and so on.) One possible cause for this error is running Arc as a local user before you set it up as a service. In this case, it is possible that certain resources created by the application were created under the local user. As a result, those resources are not available when you run Arc as a service. ### Recommended Solution In a Linux operating environment, the easiest way to ensure that the service account (or any other account that you want to use to run Arc) can access *`ApplicationDirectory`* is to use the `chown` command. For example, if *`ApplicationDirectory`* is in the default Linux location and Arc should run under the service account, the following command should resolve the error: ```bash theme={null} sudo chown -R cdatarc:cdatarc /opt/arc ``` # Defining a Custom Master Key Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/custom-master-key How to configure and rotate custom master keys for encrypting sensitive data in CData Arc. CData Arc uses a master key to secure sensitive encrypted data such as passwords, API keys, and connection credentials stored in connector configurations. By default, Arc provides a built-in master key, but organizations with strict security requirements can define custom master keys to maintain full control over their encryption infrastructure. This enables you to easily rotate user-defined master keys. Master keys can be provided in two formats: As a random character string in an AES key file or as a PFX (PKCS#12) certificate file with optional password protection. Custom master keys enable enterprises to align Arc's encryption practices with their internal security policies, comply with regulatory requirements, and integrate with existing key management systems. > [!WARNING] > > Losing access to a custom master key prevents Arc from decrypting stored credentials and configuration data. You should implement secure key backup and recovery procedures before configuring a custom master key. ## Understanding Data Encryption Keys and Master Keys Encryption works by using a *key*, which is a piece of data, to scramble plaintext into ciphertext. When you first begin using the application, Arc generates a random key that is never saved in plaintext anywhere. That is the *data encryption key*, or DEK. All data is encrypted using this key. A *master key* (also known as a Key Encryption Key or KEK) does not encrypt data directly. Instead, it is a wrapper which encrypts other keys. Arc also provides a built-in master key, but as mentioned above, you can define your own master key instead. The remainder of this topic describes how to create or update your user-defined master key. The following table compares the two keys: | | Master key | Data encryption key | | - | - | - | | Encrypts | The data encryption key | Your stored values | | Change it with | `appChangeMasterKey` (.NET) or `-ChangeMasterKey` (Cross-Platform) | [importDataEncryptionKey](../../admin-api/admin-api-actions) | | Must match for one instance to decrypt another instance's values? | No. Instances can use the same master key file or different ones. | Yes. Transfer it with [exportDataEncryptionKey](../../admin-api/admin-api-actions) and [importDataEncryptionKey](../../admin-api/admin-api-actions). | > [!NOTE] > > Using the same master key file on two instances does not let them decrypt each other's values. Only the data encryption key determines that. Do not point an existing instance's `AppKey` at a different master key file. Use the change master key command instead, or the instance fails to start with an *Incorrect user master key or key file is damaged* message. ## Check Whether Master Key Encryption Is Enabled Some features, such as the data encryption key export and import used for [development environments](../administration/git-version-control#development-environments), require master key encryption to be enabled. To check whether it is enabled on an instance: * **Cross-Platform edition:** Master key encryption is enabled if the [cdata.encryption.key](./arc-properties-configuration#cdata-encryption-key) property is set in the [arc.properties](./arc-properties-configuration) configuration file, or the `APP_KEY` environment variable is set. * **.NET edition:** Master key encryption is enabled if an `AppKey` entry exists in the `appSettings` element of the `Web.Config` file in the `www` folder of the installation directory, or the `APP_KEY` environment variable is set. New installations enable master key encryption by default. If master key encryption is not enabled, those features fail with the error *Master key encryption is not enabled*. ## Enable Master Key Encryption To enable master key encryption on an instance where it is not enabled, including an existing instance that already stores values: 1. Stop Arc. 2. Set the master key: * **Cross-Platform edition:** Set the [cdata.encryption.key](./arc-properties-configuration#cdata-encryption-key) property in the [arc.properties](./arc-properties-configuration) configuration file: ``` cdata.encryption.key=default.master_key ``` * **.NET edition:** Add the following entry to the `appSettings` element of the `Web.Config` file in **both** the `www` folder and the `www_services` folder of the installation directory: ```xml theme={null} ``` 3. Start Arc. If you use `default.master_key` and the file does not exist, Arc generates it in the application directory. Values that are already stored remain readable. In both editions, you can also use the `APP_KEY` environment variable to supply the key file path. Relative paths are resolved against the application directory. To use your own key file instead of the default, see [Define Custom Master Key](#define-custom-master-key). ## Define Custom Master Key The way in which you define a master key varies by Arc edition type. * **Cross-Platform edition:** Define the key using the [cdata.encryption.key](./arc-properties-configuration#cdata-encryption-key) property in the [arc.properties](./arc-properties-configuration) configuration file. If you are defining a PFX key and require a password, set it in the [cdata.encryption.key.password](./arc-properties-configuration#cdata-encryption-key-password) property. Alternatively, if you are using an AES key, you can use the `APP_KEY` environment variable to supply the file path to the key file. * **.NET edition:** Set a file path in the `AppKey` setting in the `appSettings` element of the `Web.Config` file in both the `www` folder and the `www_services` folder of the installation directory. If you are defining a PFX key and require a password, set it in the `AppKeyPassword` setting. Here is the XML snippet in Web.Config that you need to add: ```xml theme={null} ``` ## Change Master Key If necessary, you can replace the master key file. Arc always uses the key that you have provided for encryption. To change the master key, ensure that you have stopped the server, then use one of the following options: * **Cross-Platform edition:** Administrators can use the `arc.jar -ChangeMasterKey ` command to rotate the master key. If you are changing the master key to a PFX file, and you require a password, use the `arc.jar -ChangeMasterKey ` command. * **.NET edition:** Administrators must use the `CData.exe appChangeMasterKey -NewKey -@libdir "www\bin"` command (do not modify the Web.Config file directly). If you are changing the master key to a PFX file, and need a password, use the `CData.exe appChangeMasterKey -NewKey -NewKeyPassword -@libdir "www\bin"` command. These commands only replace the master key on an instance where [master key encryption is already enabled](#enable-master-key-encryption). If it is not enabled, they fail with the error *The encryption key is not set*. They also keep the same data encryption key, so you cannot use them to share keys between instances. To share the data encryption key, use [exportDataEncryptionKey and importDataEncryptionKey](../../admin-api/admin-api-actions). > [!NOTE] > > Arc only supports the RSA algorithm in PFX certificate files. Other DSA or ECC certificates are rejected. ## Use Safe Mode If you lose your custom master key and need to re-save your configuration's sensitive values (from another source of record that you have), you can start Arc in safe mode. This disables all backend automation services and enables very limited features so you can try to recover your settings and other tasks. To enable safe mode: * **Cross-Platform edition:** Run `java -jar arc.jar -safemode` * **.NET edition:** Add `` to the `appSettings` element of your Web.Config file. # How to Upgrade CData Arc Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/how-to-upgrade Step-by-step instructions for upgrading CData Arc from various previous versions. CData Arc 2026 is the latest version of several earlier application releases, including the following: * CData Arc 2022 and 2023 * ArcESB 2020 and 2021 * RSSBus Connect 2016, 2017, 2018, and 2019 * AS2 connector V2 (/n software) * AS2 connector V3, FTP connector V1, OFTP connector V2, and SFTP connector V1 (RSSBus) * IP\*Works! EDI AS2 connector V7 (/n software) > [!NOTE] > > Download the latest software version from \[[https://arc.cdata.com/support/builds/](https://arc.cdata.com/support/builds/)]\([https://arc.cdata.com/support/builds/](https://arc.cdata.com/support/builds/)). > > Upgrading to a new major version (for example, upgrading from Arc 2023 to Arc 2026) requires new product keys to license the application. You can request new keys from our sales team at \[[arcsales@cdata.com](mailto:arcsales@cdata.com)]\(mailto:[arcsales@cdata.com](mailto:arcsales@cdata.com)). ## Upgrading from a Build of CData Arc 2024 or Later ### Cross-Platform Edition The Java Edition of Arc 2026 is shipped as setup.exe for Windows and in a compressed setup.tar.gz file for Linux. Follow the steps for your operating system to upgrade an existing installation of Arc 2026 to a newer build. #### Linux 1. Stop the Arc server. 2. Unzip the `arc.war` file from the tar.gz archive and place it in `/opt/arc/webapp/`, overwriting the old file. 3. (Optional) Unzip the `arc.jar` file from the tar.gz archive and place it in `/opt/arc/`, overwriting the old file. 4. Start the server. #### Windows 1. Stop the Arc server. 2. Run setup.exe. 3. Point the installer to the location where Arc is installed. 4. After the upgrade completes, start the server. #### Notes for Java users * No matter how you choose to deploy Arc, the files and folders that are contained in the application directory (*`ApplicationDirectory`*) must be readable and writable by the user running the application. The service.sh script that is included with the Java Edition uses **cdatarc** as this user. * If the application was previously run as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, the command should be similar to the following: ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` ### .NET Edition Stop the server (either through the embedded web server UI or in IIS, if applicable) and run setup.exe. The setup detects the installation path, overwrites the older build, and leaves all configuration data intact. ## Upgrading from CData Arc 2022 or 2023 ### Cross-Platform Edition The Java Edition of Arc 2026 is shipped as setup.exe for Windows and in a compressed setup.tar.gz file for Linux. Follow the steps for your operating system to upgrade an existing installation of Arc 2022 or 2023 to Arc 2026. #### Linux 1. Stop the Arc server. 2. Unzip the `arc.war` file from the tar.gz archive and place it in `/opt/arc/webapp/`, overwriting the old file. 3. (Optional) Unzip the `arc.jar` file from the tar.gz archive and place it in `/opt/arc/`, overwriting the old file. 4. Start the server. #### Windows 1. Stop the Arc server. 2. Run setup.exe. 3. Point the installer to the location where Arc is installed. 4. After the upgrade completes, start the server. #### Notes for Java users * No matter how you choose to deploy Arc, the files and folders that are contained in the application directory (*`ApplicationDirectory`*) must be readable and writable by the user running the application. The service.sh script that is included with the Java Edition uses **cdatarc** as this user. * If the application was previously run as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, the command should be similar to the following: ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` ### .NET Edition Arc 2026 functions as a drop-in upgrade to Arc 2022; simply running the Windows installer for Arc 2026 upgrades all necessary application resources. The Windows installer for Arc 2026 attempts to detect existing installations of ArcESB 2022 and preserve the connectors, flows, and settings from the existing installation. If ArcESB 2022 was not installed in the default location, you might need to point the Arc 2026 installer to the folder on disk where ArcESB 2022 was installed. ## Upgrading from ArcESB 2021 ### Cross-Platform Edition The Java Edition of Arc 2026 is shipped as setup.exe for Windows and in a compressed setup.tar.gz file for Linux. Follow the steps for your operating system to upgrade an existing installation of ArcESB 2021 to Arc 2026. #### Linux 1. Extract the tar.gz file to `/opt/arc/`. 2. Stop the server (for example, `sudo systemctl stop arcesb`). 3. If upgrading an installation that utilizes the Embedded Jetty Server, run `sudo bash /opt/arc/upgrade.sh`, optionally specifying the `-InstallDirectory` parameter if your previous installation used a non-default path. If upgrading an installation that utilizes an external servlet container (for example, Apache Tomcat or External Jetty), see the [instructions for your servlet container](./cross-platform-edition). 4. Start the server either as a service (for example, `sudo systemctl start arc`) or in-process (for example, `java -jar arc.jar`). #### Windows 1. Stop the Arc server. 2. Run setup.exe. 3. Point the installer to the location where Arc is installed. 4. After the upgrade completes, start the server. #### Notes for Java users * No matter how you choose to deploy Arc, the files and folders that are contained in the application directory (*`ApplicationDirectory`*) must be readable and writable by the user running the application. The service.sh script that is included with the Java Edition uses **cdatarc** as this user. * If the application was previously run as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, the command should be similar to the following: ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` ### .NET Edition Arc 2026 functions as a drop-in upgrade to ArcESB 2021; simply running the Windows installer for Arc 2026 upgrades all necessary application resources. The Windows installer for Arc 2026 attempts to detect existing installations of ArcESB 2021 and preserve the connectors, flows, and settings from the existing installation. If ArcESB 2021 was not installed in the default location, you might need to point the Arc 2026 installer to the folder on disk where ArcESB 2021 was installed. ## Upgrading from ArcESB 2020 Before you begin, keep in mind that the current version of Arc includes several features that might cause the application to behave differently from an ArcESB 2020 installation: * Arc 2026 manages users within the application rather than through server configuration, so some user settings might not transfer. See [User Management](#user-management) for more information. * Arc 2026 does not automatically preserve subfolder structures found on FTP and SFTP servers. See [Subfolders in File Based Connectors](#subfolders-in-file-based-connectors) for more information. * Arc 2023 does not let you override local folders for individual connectors. See [Overriding Local Folders](#overriding-local-folders) for more information. * Arc 2026 does not require Windows administrator privileges to run by default, so some folder permissions might need to be updated. See [Windows Application Directory](#windows-application-directory) for more information. ### Cross-Platform Edition The Java Edition of Arc 2026 is shipped as setup.exe for Windows and in a compressed setup.tar.gz file for Linux. Follow the steps for your operating system to upgrade an existing installation of ArcESB 2020 to Arc 2026. #### Linux 1. Extract the tar.gz file to `/opt/arc/`. 2. Stop the server (for example, `sudo systemctl stop arcesb`). 3. If upgrading an installation that utilizes the Embedded Jetty Server, run `sudo bash /opt/arc/upgrade.sh`, optionally specifying the `-InstallDirectory` parameter if your previous installation used a non-default path. If upgrading an installation that utilizes an external servlet container (for example, Apache Tomcat or External Jetty), see the [instructions for your servlet container](./cross-platform-edition). 4. Start the server either as a service (for example, `sudo systemctl start arc`) or in-process (for example, `java -jar arc.jar`). #### Windows 1. Stop the Arc server. 2. Run setup.exe. 3. Point the installer to the location where Arc is installed. 4. After the upgrade completes, start the server. #### Notes for Java users * No matter how you choose to deploy Arc, the files and folders that are contained in the application directory (*`ApplicationDirectory`*) must be readable and writable by the user running the application. The service.sh script that is included with the Java Edition uses **cdatarc** as this user. * If the application was previously run as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, the command should be similar to the following: ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` ### .NET Edition Arc 2026 functions as a drop-in upgrade to ArcESB 2020. Simply running the installer for Arc 2026 upgrades all necessary application resources. The installer for Arc 2026 attempts to detect existing installations of ArcESB 2020 and preserve the connectors, flows, and settings from the existing installation. If ArcESB 2020 was not installed in the default location, it might be necessary to point the Arc 2026 installer to the folder on disk where ArcESB 2020 was installed. Arc 2026 can be installed separately from ArcESB 2020 by directing the Arc 2026 installer to a different folder than the one in which ArcESB 2020 was installed. Installations do not interfere with each other as long as they are installed to different folders. ### User Management ArcESB 2020 and earlier managed users via the server configuration XML files (for example, Web.Config or server.xml). More recent versions of Arc bring user management within the application itself, on the **Users** tab in the **Settings** page. Users configured in versions prior to 2021 do not persist during the upgrade and must be re-created as necessary on the **Users** tab. See [Users](../administration/settings/users) for more information. ### Subfolders in File Based Connectors > [!NOTE] > > This section is relevant only for users with file-based connectors (including File, FTP, and SFTP) that preserve the folder structure from remote file systems. Previous versions of Arc exhibited the following behavior: These file-based connectors target a single folder on a remote server from which to upload or download files. When files should also be downloaded from subfolders within this target folder, this same subfolder structure was re-created locally in the connector's **Transactions** tab. Likewise for uploads, these connectors treated a file being in a subfolder of the **Transactions** tab as an indication of the subfolder the file should be placed in on the remote file system. This recreation of subfolder structure is what is meant by "preserving subfolder structures". Arc 2026 requires the use of the File connector to mirror folder structures on remote systems. These file based connectors add a **subfolder** header to downloaded files if the file was stored in a subfolder of the target folder on the server. This **subfolder** header is used to automatically reproduce the subfolder structure when files are uploaded to remote file systems. ### Overriding Local Folders Prior to ArcESB 2021, you could override the `Send`, `Receive`, and `Sent` folders for individual connectors by using **Connector Settings**. Starting with ArcESB 2021, these options are not in the interface, and direct overrides of connector folders are discouraged. Best practice is to [use a File connector](../../flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems. If necessary, you can still manually override the SendFolder, ReceiveFolder, and SentFolder properties of connectors in **Connector Settings** by using the field under **Advanced** > **Miscellaneous** > **Other Settings**. If these settings were overridden for a connector in a previous release, the configured settings are present here. If you choose to use this method to override local folders for the connector, set the value under **Advanced** > **Processing Delay** to a non-zero value in seconds (for example, 10) to prevent Arc from processing files that are still being written by an external process. ### Windows Application Directory The Windows edition of Arc 2026 stores the Application Directory (which holds the data for flows, connectors, profile settings, and so on) in the `C:\ProgramData\CData\Arc` folder by default. Prior to ArcESB 2021, the Application Directory was in the `C:\Program Files` folder. This helps ensure that Arc 2026 can run without administrator privileges by default. When you upgrade from ArcESB 2020 or earlier to Arc 2026, the application re-uses the Application Directory from the older installation, but it still attempts to run without administrator privileges by default. As a result, it might be necessary to set the access permission rules on the Application Directory within `C:\Program Files` to allow Arc access with lower running permissions. For more information on the Application Directory, see [Configuring the Application Directory](./windows-edition#configuring-the-application-directory). ## Upgrading from RSSBus Connect 2016, 2017, 2018, or 2019 This section describes the process of upgrading from an earlier release of RSSBus Connect. > [!NOTE] > > For help upgrading from an earlier version than RSSBus Connect 2016, see [Upgrading from AS2 Connector](#upgrading-from-as2-connector). Before you begin, keep in mind that the current version of Arc includes several features that might cause the application to behave differently from an RSSBus Connect installation: * Arc 2026 manages users within the application rather than through server configuration, so some user settings might not transfer. See [User Management](#user-management) for more information. * Arc 2026 does not automatically preserve subfolder structures found on FTP and SFTP servers. See [Subfolders in File Based Connectors](#subfolders-in-file-based-connectors) for more information. * Arc 2026 does not let you override local folders for individual connectors. See [Overriding Local Folders](#overriding-local-folders) for more information. * Arc 2026 does not require Windows administrator privileges to run by default, so some folder permissions might need to be updated. See [Windows Application Directory](#windows-application-directory) for more information. ### Cross-Platform Edition The Java Edition of Arc 2026 is shipped as setup.exe for Windows and in a compressed setup.tar.gz file for Linux. Follow the steps for your operating system to upgrade an existing installation of RSSBus Connect Java Edition to Arc 2026: #### Linux 1. Extract the tar.gz file to `/opt/arc/`. 2. Stop the server (for example, `sudo systemctl stop arcesb`). 3. If upgrading an installation that utilizes the Embedded Jetty Server, run `sudo bash /opt/arc/upgrade.sh`, optionally specifying the `-InstallDirectory` parameter if your previous installation used a non-default path. If upgrading an installation that utilizes an external servlet container (for example, Apache Tomcat or External Jetty), see the [instructions for your servlet container](./cross-platform-edition). 4. Start the server either as a service (for example, `sudo systemctl start arc`) or in-process (for example, `java -jar arc.jar`). #### Windows 1. Stop the Arc server. 2. Run setup.exe. 3. Point the installer to the location where Arc is installed. 4. After the upgrade completes, start the server. #### Notes for Java users * No matter how you choose to deploy Arc, the files and folders that are contained in the application directory (*`ApplicationDirectory`*) must be readable and writable by the user running the application. The service.sh script that is included with the Java Edition uses **cdatarc** as this user. * If the application was previously run as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, the command should be similar to the following: ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` #### .NET Edition Upgrading to Arc 2026 from RSSBus Connect 2016, 2017, 2018, or 2019 can be a one-step in-place upgrade. Simply run the Arc installer to update application resources (EXE, DLL, JAR, and WAR files) without disturbing configuration data. If the Arc installer detects an RSSBus Connect installation, the installer uses this same installation path to store configuration data (profile, connectors/ports, certificates). Thus, the settings from older versions are used automatically by the new Arc installation. ### Transferring Configuration Data Manually If an in-place upgrade is not possible, you can still transfer configuration data from previous versions of RSSBus Connect. The first transfer method is to export connector (port) settings from the older version, then import these connectors into the new version: 1. In RSSBus Connect, export connectors from the **Ports** tab by clicking **Settings** (the gear icon) in the upper right of the window and selecting **Export Port Settings**. In later versions of RSSBus Connect (for example, RSSBus Connect 2019), you can export connectors from the **Flows** tab by right-clicking the connector (port) in the flow and selecting **Export Port Settings**. 2. In Arc, open the workspace's **More Options** ellipsis and select **Import Flows** to import the connectors. The second method transfers all configuration data at once by managing folders on disk. All configuration information is held in the `data` folder of the installation directory. Copying the `data` folder from a previous version into the same folder in the latest version effectively copies the entire application configuration. The default installation directory for Arc is as follows: * **.NET Edition**: `C:\Program Files\CData\CData Arc` * **Java**: `/opt/arc` The default installation directory for RSSBus Connect is as follows: * **.NET Edition**: `C:\Program Files\RSSBus\RSSBus Connect` * **Java**: `~/rssbus/connect` On a Linux system, `~` resolves to the home directory of the user where the Java servlet is installed. On a Windows system, `~` resolves to `%ProgramData%`. > [!NOTE] > > If you do not see the Arc installation directory, check for the RSSBus Connect default installation directory. The Arc installer attempts to detect RSSBus Connect installations and use the pre-existing folder structure from RSSBus Connect. ### Upgrading Application-Data Source Connectors Arc introduced a new approach to connecting to external data sources that are applications rather than traditional databases (for example, Salesforce, NetSuite, QuickBooks, and so on). In RSSBus Connect, each external data source application required a dedicated connector (port) to interface with the application. In Arc, a single connector called the [CData connector](../../connectors/cdata/cdata) is used to connect to each application data source. #### CData Drivers The CData connector uses CData drivers to connect to a specific data source. [CData Software](https://www.cdata.com/) provides the ADO.NET and JDBC drivers that enable connectivity to hundreds of cloud and on-premises applications, databases, and more. These drivers were used beneath the surface in previous RSSBus Connect connectors (ports), so the interface and approach to accessing data in these application data sources has not changed. To upgrade this set of connectors (ports) to Arc, you must install and configure the appropriate CData driver in a CData connector. See the [CData Connector](../../connectors/cdata/cdata) documentation for details on installing CData drivers. Once the CData driver is installed and selected within the CData connector's settings, the CData connector can be configured with the same connection settings and credentials that were used in RSSBus Connect connectors (for example, the Salesforce connector, the HubSpot connector, and so on). ### Upgrading External Connectors A small number of connectors that were previously external are now embedded in Arc 2026: * SQL Server * Oracle * PDF These connectors are installed appropriately along with the rest of the application, and no longer require additional downloads. ## Upgrading from AS2 Connector This section also applies to upgrading from **FTP Connector**, **OFTP Connector**, and **SFTP Connector**. ### Installing CData Arc #### .NET Edition Users The Windows installation of Arc targets a different installation path than earlier versions of the product, and it listens on a different port. This change ensures that both releases can be run side-by-side without conflicting with one another. Simply run setup.exe and follow the prompts to install the application on the machine. #### Java Users As with previous versions of the AS2 connector (and other releases in the connector family), the application is provided as a distributable `arc.war` file. If you distribute this WAR file to your Java server by using the server's built-in deployment wizard, this replaces your existing installation of the AS2 connector in its entirety, including trading partner profiles. If you want to perform an in-place upgrade of the AS2 connector and still preserve your profile settings between installations, create a copy of your previous installation's `data` folder before you deploy the new package. This `data` folder is located in `/WEB-INF/data/` under the existing web application folder. Once you copy this folder into a new location, deploy Arc in place of the existing installation. When you launch the application for the first time, the connectors (ports) are not configured. Click **Import Profiles** to specify the location where you copied your `data` folder to restore your profiles. Alternatively, you can rename the `arc.war` file that is provided with Arc to install this version of the application as a parallel application. ### Importing Profile Information If you are upgrading from an earlier version of the AS2 connector, SFTP connector, or other previous versions, you can detect and import your existing partner profile configurations using two methods. #### Import Automatically If you install the Windows edition of Arc on a machine where a previous version of the application was installed, the installation program attempts to detect the installation of the previous version, and it can import the profile settings from it. If it detects a partner profile with the same name as one already configured in Arc, this process creates a duplicate entry with a number appended to the identifying field to ensure that it is unique. You can safely delete duplicate profiles after they are imported. #### Import Using the Admin API Arc's Admin API includes an *import* action that you can use to import settings from previous versions. The **import** action requires three parameters: * **arcflow** — The base64-encoded string containing the flow data. This string can be generated by base64-encoding the entire file contents of a `.arcflow` file. * **Overwrite** — How Arc should respond if an imported profile/partner configuration causes a conflict with the current configuration. If the value is **True**, the existing configurations are overwritten. Otherwise, an error is generated. * **WorkspaceId** — The workspace into which the flow is imported. If this parameter is not specified, the application attempts to determine which workspace to import into (based on workspace values in the flow that is being imported) and falls back to the default workspace if necessary. All requests to the **import** action must be sent as HTTP POST requests. For more information about the Admin API, see [Admin API](../../admin-api). ### Configuring a Transparent Upgrade After you run the Arc installer and import partner profiles from a previous installation, the new version has backwards compatibility for receiving URLs in the old version. However, there are two key differences: * Arc uses a different local port than the application server that hosted the previous installation of the application. This behavior avoids conflicts with the installation of a previous version of the AS2 connector so that you can maintain both versions of the application and transfer partners to the new version at your own pace. * The `Incoming`, `Outgoing`, and `Sent` folders from the previous installation of the AS2 connector are not imported automatically into the profiles that are used in Arc. By default, the application uses its own local file resources, so it does not interfere with the currently running instance of the application. This also ensures that, if automation is enabled in the previous profile, the application does not immediately begin sending files that are waiting to be sent in the older version of the AS2 connector. To replace your instance of the AS2 connector: 1. Determine whether to use the old folder structure from the previous installation of the AS2 connector, or the Arc default folder structure. 2. Determine whether to use your existing receiving URL, or a new one. #### Keeping Your Previous Folders The current version of Arc includes a File connector that lets you interact with external folders in the file system as a separate logical step in your message flow. This process lets you integrate with the local file system by using the same resources that you used previously. However, if you add additional connectors in the future, the application creates these in a different `data` folder by default. The following image shows an example of a flow that uses File connectors in this way: ![Flow using File connectors to interact with existing folders](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/file_as2_flow.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=540dac10520de1e462b7b1add7f1d732) See [Interacting with the Local File System](../../flows#interacting-with-the-local-file-system) for information about how to configure File connectors for this purpose. #### Advantages The existing back-end business logic is preserved for configured trading partners. #### Disadvantages Additional connectors use a different `data` folder by default. #### Switching to Use the New Folders If you plan to add more partners or configure additional connectors in Arc, it might be best to switch to the new default folder structure. This structure is described below. The Arc Application Directory contains all connector and profile configuration data. Each configured connector in Arc has its own folder within the `data` folder (for the default workspace) or an appropriate workspace folder within the `workspaces` folder. The connector-specific folder contains the following folders: * `Logs` — A folder where the connector writes connector-specific logs. * `Pending` — Connectors that wait for a partner response store pending messages in this folder (only certain connector types require this). * `Receive` — Connectors that receive or download files from an external source place those files in this folder. Connectors that operate on local data (without sending or receiving from an external source) place files here after they are processed. * `Send` — Files placed in this folder are slated to be processed, sent, and uploaded by the connector. * `Sent` — The connector places a copy of processed, sent, and uploaded files in this folder. For AS2 connectors, place files that should be sent to external trading partners in the connector's `Send` folder, and pick up files that are received from external trading partners in the connector's `Receive` folder. #### Advantages A consistent folder structure is maintained if additional trading partners are configured. #### Disadvantages Business logic might need to be reconfigured to use the new folder structure. #### Using a New URL The server provided with Arc listens on different ports and uses different URLs for receiving AS2 transmissions than earlier releases. After you import your profiles into Arc, you must update your trading partners so that they are sending to your new Receiving URL. By default, the server provided with Arc listens in plain text on port 8001 and uses SSL/TLS on port 8401. You can find the Receiving URL by clicking **Profile > AS2**. #### Advantages * Trading partners transition at their own pace. * There is no downtime. #### Disadvantages * Trading partners need to change your URL in the profile configuration on their end. * An additional port must be opened on the firewall. * Both versions of the application are maintained. #### Using the Existing URL To receive requests on the same URL used by the previous version, stop the server hosting the previous version so that the same port can be assigned to Arc. You also need to assign the server to the port that you were using for the previous version. You can configure the port by right-clicking the icon in the system tray, then clicking **Server Options**. The application automatically recognizes requests to resources that were supported in previous versions of the application (`/as2connector/pub/ReceiveFile.rsb` and `/Receiver.aspx`, for instance) and forwards them to the appropriate resources in Arc. #### Advantages * Trading partners do not need to change profiles. * Maintain one system. #### Disadvantages * Downtime while old server is stopped to free up a port for the new server. * Existing profiles must first be imported. > [!NOTE] > > If the port that you configure is an SSL/TLS-enabled port, you also need to configure the SSL/TLS certificate that is used for the previous version of the AS2 connector on this port. If you previously hosted an SSL/TLS port in Arc, the same certificate store is the location of SSL/TLS certificates in Arc (the local machine's personal account). After you configure Arc to use the same port that you used in older versions of the application, you start receiving incoming transmissions at the folders configured above. ## Upgrading an Amazon AMI Arc 2026 has a dedicated AMI in the AWS Marketplace that you can use to create a new EC2 instance running Arc. To preserve all application data, follow the instructions in Manual Backup to understand how to copy application folders directly from the file system. Follow these steps to make a persistent copy of the relevant folders: 1. Copy the folders from the Amazon VM to a local machine via a transfer protocol like SFTP or SCP. 2. Transfer the same folders to the EC2 instance created from the latest AMI. # Installation and Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/installation-configuration System requirements, browser support, and high-level configuration topics for CData Arc. CData Arc is available in both a .NET and Java edition. The Java edition provides cross-platform support including Linux and Mac. Both editions come with an embedded Web server, but can also be deployed to other Web servers for their respective platforms. ## System Requirements Your machine must meet the minimum system requirements listed below to run Arc. ### .NET * Windows 10/Windows Server 2016 or higher. * .NET Framework 4.6.2 or higher. * 2 GB RAM recommended (4+ GB for higher volumes of traffic). * Adequate free disk space for transmission logging and writing received files. Use of an HDD drive is not recommended except for low volumes of activity. > [!NOTE] > > Arc functions on earlier versions of Windows, but Windows operating systems that have reached end-of-life might have limited support. ### Cross-Platform Edition * Java 17 or later is required. * 2 GB RAM recommended (4+ GB for higher volumes of traffic). By default, Arc uses 60% of the memory available. You can adjust that in scenarios where the total amount of RAM is low, or where Arc is running on a server with other memory-intensive applications. * Adequate free disk space for transmission logging and writing received files. Use of an HDD drive is not recommended except for low volumes of activity. ### Internet Connection If the application must connect with external systems, you might need the following resources: * A persistent Internet connection. * An external IP address for the system running the application. * Firewall/proxy access to communicate with the external system. ## Supported Web Browsers The Arc web UI is accessed through a web browser. Arc supports the following browsers: * Google Chrome * Mozilla Firefox * Safari * Microsoft Edge Other browsers might not support all Arc features. As a result, CData strongly recommends using a browser from the list above. ## Application Database If you expect high transaction volumes, choose an external database before you install. Migrating from the default embedded database to an external one after the fact requires additional steps. ### .NET By default, Arc uses a SQLite database in the installation directory as the application database. This database is recommended for up to 100,000 transactions. Once you hit that number, CData recommends moving to an external database. You can configure the application to use an enterprise database like SQL Server, PostgreSQL, or MySQL. For configuration steps, see [Configuring the Application Database](./windows-edition#configuring-the-application-database). ### Cross-Platform Edition By default, Arc uses an H2 database that resides in *`ApplicationDirectory`* as the application database. This database is recommended for up to 100,000 transactions. Once you hit that number, CData recommends moving to an external database. You can configure the application to use an enterprise database like SQL Server, PostgreSQL, or MySQL. For configuration steps, see [Configure the Application Database](./cross-platform-edition#configure-the-application-database). ## Clustering The [Clustering](../administration/clustering) topic describes how to set up multiple instances of Arc to process the same data and use the same configuration. This allows for a load-balanced or high-availability setup. ## DMZ Gateway The [DMZ Gateway](../administration/dmz-gateway) topic explains how Arc can receive secure connections through a DMZ gateway via the use of an SSH reverse tunnel. This prevents the need to expose your internal network to outside traffic. ## Split-Plane Deployment The [Split-Plane Deployment](../administration/split-plane-deployment) topic explains how Arc can host public receiving endpoints on different ports than the main administration console. This adds a layer of security redundancy to prevent external parties from gaining access to the administration console. ## Security Best Practices For information on security-related features available in Arc, and to learn about the best practices we recommend you follow, see [Security Best Practices](../administration/security-best-practices) in the [Administration](../administration/administration) documentation. # Starting the Server Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/starting-server How to start the CData Arc server on Windows, Linux, and via Java servlet containers. CData Arc is built on top of an application framework that provides web services, authentication, encryption, and automation facilities. The application can be hosted on any Windows-based or Java-based web server. Both the Windows and Cross-Platform editions include a stand-alone server, so no external server is required to host the application. ## Cross-Platform Edition The Cross-Platform edition comes bundled with a web server, built on top of Eclipse Jetty, to support simple cross-platform deployment. This stand-alone web server is launched by running the `arc.jar` file, which is located in the installation directory. ``` java -jar arc.jar ``` Arc comes pre-configured to work immediately in any environment. However, if you need to customize the configuration for this web server you can [generate the arc.properties file](./cross-platform-edition#generating-the-arc-properties-file), which is located next to the `arc.jar` file. Once it has been created you can modify this configuration file according to your specific deployment needs. In addition, Arc provides a `service.sh` script that you can use to set up a **systemd** or **init.d** service on Linux systems. You can also deploy Arc to any Java servlet container. See [Cross-Platform Edition](./cross-platform-edition) for step-by-step deployment guides to several major Java web servers. ### Microsoft Windows In Windows, Arc is installed as a service by default. To access the application, you must first ensure that the service is running. Once the service is running, you can access the Arc admin console by opening a web browser and entering `http://localhost:8080/` in the URL field. ### Linux Service Configuration Follow these steps to ensure that your Linux installation of Arc functions as expected: 1. When you install Arc, ensure that the user who is performing the installation has read and write permissions for the install directory. The root user works fine for this. 2. Run the `service.sh` script. This script is located at the root of the setup archive, so if it was extracted to `/opt/arc`, the path to this script is `/opt/arc/service.sh`. 3. After you run the `service.sh` script, start the service according to the service manager that your Linux distribution uses: * For **init.d**, submit this command: `sudo service arc start` * For **systemd**, submit this command: `sudo systemctl start arc` > [!NOTE] > > No matter how you choose to deploy Arc, the files and folders within *`ApplicationDirectory`* must be readable and writable by the user who is running the application. The service installer that is included with the Java edition setup uses **cdatarc** as this user. If the application was run previously as a different user and you want to restore the necessary permissions for a **cdatarc** user to run the application, run this command (changing the directory if needed): ``` sudo chown -R cdatarc:cdatarc /opt/arc ``` ## .NET Edition The .NET edition comes with an embedded web server. To start up and run the stand-alone server, follow these steps: 1. Launch the server (if it is not already running) either through the Windows **Start** Menu or by running `CData.Arc.exe` in the installation directory. 2. If the web UI does not automatically open in a browser, manually launch the UI in one of the following ways: * Double-click the CData Arc icon in the system tray. * Right-click the CData Arc icon and select **Start Application**. * Use a browser to manually navigate to the address and port where the server is listening (the default port is 8080). See [Using the Embedded Server](./windows-edition#using-the-embedded-server) for information on enabling SSL/TLS and other configuration options for the embedded web server. See [Using IIS](./windows-edition#using-iis) for information about hosting Arc in IIS. # .NET Edition Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/installation/windows-edition How to host CData Arc using the embedded server or IIS on Windows, including TLS/SSL, Windows Authentication, and application configuration. The .NET edition of CData Arc includes a stand-alone embedded web server and can also be hosted in IIS. This topic describes various options for hosting the application in Windows. If you do not have experience with IIS, CData recommends you use the embedded server. The embedded server is automatically deployed at installation and provides a simple interface for server-level administration tasks such as configuring logging, running the application as a service, and enabling TLS/SSL. ## Using the Embedded Server This section shows how to configure the stand-alone, embedded Web server, including how to host SSL/TLS connections and configure the [Cloud Gateway](#cloud-gateway), a reverse SSH tunnel. When hosted on the embedded server, the application can be run as a Windows service: 1. Right-click the CData Arc icon in the taskbar and click **Server Options**. 2. Select **Run as a Windows Service**. 3. Save your changes. > [!NOTE] > > If you need to install CData drivers on the [Connections](../administration/settings/connections)**Settings** tab, and you encounter a permission error during the installation, grant `write` permission to `www\Web.Config` and `www\bin` for the Arc service account (Network Service by default). ### Host TLS/SSL Connections (Embedded Server) You can require TLS as well as offer services in plain text. This section shows how to host TLS/SSL connections in the embedded server included with the Windows edition of the application. To deploy TLS/SSL for the Java edition, refer to the documentation for the Java servlet you are using to host the application. 1. Right-click the icon for the application in the Windows system tray and click **Server Options**. 2. On the Web Server tab, select **Enable TLS** and select a port that you want to listen on for TLS connections. 3. Click the button next to the **TLS Server Certificate** box to select a private key certificate. The certificate with private key must be installed to the Local Computer certificate store. If there are no certificates available that match that criteria, click **Create New Certificate** to create a simple certificate for this purpose. ![Server options dialog for TLS configuration](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/ServerDialog.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=a622bc8d6b61cb22688e4fdd81cd7e9e) The server restarts and begins serving TLS requests after you save your changes. #### Manage Personal Certificates The certificates available in the certificate selection dialog are located in the Personal certificate store for the local computer. To add a certificate to the Windows store: 1. Type **mmc** from the Run menu to launch Microsoft Management Console. 2. Click **File > Add/Remove Snap-In**. 3. Select **Certificates** from the Available snap-ins table and click **Add**. ![Add snap-in dialog in MMC](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/AddSnapIn.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=14004c3dbe1162e160c4a3923e2a2e1c) 4. Select **Computer Account > Local Computer**. 5. After adding the snap-in, click **Certificates (Local Computer) > Personal > Certificates** to display the available certificates. The certificates for which a private key is available have a key icon superimposed over the certificate icon. 6. To import a certificate from the local disk into this store, click **Action > All Tasks > Import**. After completing the import wizard, the certificates are available in the application certificate selection dialog. ### Cloud Gateway The Cloud Gateway provides a simple way to expose the application to the internet without the need for firewall changes. As long as a publicly accessible SSH server is available, Arc can be accessed via a port on the publicly accessible SSH server by leveraging an SSH reverse tunnel. To leverage this functionality, right-click the application icon in the system tray and select **Server Options**. Navigate to the **Cloud Gateway** tab, as shown below, and enter the following information: ![Cloud Gateway configuration tab](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/CloudGateway.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=75c59d63c3e05bbacec4aebbb83010f4) | Setting | Description | | - | - | | **Host** | The SSH server that Arc connects to. | | **Port** | The port on which the SSH host is listening for connections. Most SSH servers listen on port 22, which is the default value. | | **Authentication Type** | The type of authentication to use. Arc supports Password, Public Key, Multi-Factor, and Keyboard Interactive. | | **User** | The username Arc uses to authenticate to the SSH server. | | **Password** | The password used to authenticate, if required by the selected **Authentication Type**. | | **Certificate** | The certificate Arc uses to authenticate to the SSH server when using Public Key authentication. | | **Server Fingerprint** | The SSH host key fingerprint of the SSH server. This value is read-only and purely informational. | | **Forwarding Port** | The port that the publicly available SSH server listens on for connections to forward to Arc. | In addition, you might need to change the configuration of your SSH Server to allow port forwarding. Default SSH server configuration might not always enable port forwarding to non-localhost addresses. For example, if you are connecting to an OpenSSH server, the **GatewayPorts** property must be set to **yes** or **clientspecified** in the SSH server configuration file. After you enter this information, click **Test Connection** to test the connection to the SSH server to verify the supplied information. If successful, once the application is restarted, Arc is accessible via the **Forwarding Port** on the SSH host. For example, if you specify the SSH server *my.ssh.host* with a **Forwarding Port** of 8401, entering `https://my.ssh.host:8401` in a browser causes the communication to be forwarded to the local machine where Arc is running, allowing public access even though it is not directly accessible. ### Windows Authentication By default, the embedded web server uses ASP.NET's forms-based authentication, which requires that a username/password combination be entered into a webform (a login portal) to grant access to the administration console. Arc also supports Windows Authentication to grant application access to specific Windows users or security groups. Before enabling Windows Authentication on the embedded web server, you should add all [Users](../administration/settings/users) who need access to Arc to the Arc application. All Windows AD Arc user names must use the following syntax: *DOMAIN\Username*. The syntax is shown in the following image. ![Add AD user dialog showing DOMAIN\Username format](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/add_ad_user.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=b37d250c6982e64991e5507e1fb3f11c) > [!NOTE] > > You must add users individually, because an entire group cannot be granted access to the application. Next, to enable Windows Authentication, open the embedded web server configuration UI (right-click the Arc icon in the system tray and select **Server Options**), then toggle the **Authentication Type** field under the **Web Server** tab. Click **Save Changes**. This restarts the web server and the application. ![Authentication type toggle in Server Options](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/windows_auth_type.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=2923bbf9e3c1d857faf468bfd918a9b5) From then on, when Windows AD users who have been added to Arc log onto their machines and navigate to Arc in the browser, they are automatically logged in. ## Using IIS Arc supports IIS 10, and the [System Requirements](./installation-configuration#system-requirements) specify that your machine must be running Windows 10 or Windows Server 2016 or higher. If you run into difficulties setting up IIS, see the [Troubleshooting IIS](#troubleshooting-iis) section. If you still have questions, contact our [support team](https://arc.cdata.com/support/submit.aspx) for further assistance. #### Create a New Web Application > [!NOTE] > > This example uses the default website. To host the application on your website: 1. In the IIS Manager, expand the node for your server in the **Connections** panel. 2. Expand the Sites node, right-click your website, and click **Add Application**. 3. In the **Add Application** dialog that is displayed, enter the following information: | Setting | Description | | - | - | | **Alias** | The name for the application. For example, `arc`. | | **Application Pool** | The application pool associated with the application. This example uses `DefaultAppPool`. | | **Physical Path** | The path to the www directory, in the directory where the application is installed. The default location is `C:\Program Files\CData\CData Arc\www`. | #### Configure Directory Permissions The application must have full access to the Arc application directory for full functionality. In ArcESB 2021 and earlier, the application directory and the installation directory were the same: `C:\Program Files\ArcESB\` by default. If you upgraded from a previous version of Arc to Arc 2026, this is likely still your application directory. In Arc 2021, the application directory was split from the install directory. The installation directory is now `C:\Program Files\CData\CData Arc`. The application directory is now `C:\ProgramData\CData\Arc\`. The application directory contains the following folders: * connectors * data * db * locks * logs * schemas * workspaces The install directory contains: * Program executables (`CData.Arc.exe` and `arc.exe`) and configuration files * the `www` folder and subfolders * the `www_services` folder and subfolders To set permissions, locate the application directory for your Arc installation and follow these steps: 1. Right-click the folder and click **Properties**. On the Security tab, click **Edit > Add**. 2. In the **Enter the object names to select** box, enter the following, substituting the name of the application pool: `IIS AppPool\[your-application-pool]`. For example, `IIS AppPool\DefaultAppPool`. 3. Ensure that the application pool has the following permissions: * Read * Write * Modify * Read & Execute * List Folder Contents > [!NOTE] > > You can also use the command line to allow access to the application pool. For example: `icacls "" /grant "IIS APPPOOL\DefaultAppPool":(OI)(M)` #### Configure Server Permissions If you are using sftpserver or ftpserver, you must ensure that the application pool has the correct permissions set to the root directory that you have configured for the server. If you created the root directory as a child of the application directory path, setting permissions on the parent application directory and its subfolders should be sufficient. #### Prevent Application Processes from Unloading IIS can shut down a web application for several reasons, including when an idle timeout is exceeded, or when it deems the application pool's resource usage is too high. This might prevent background tasks from running in your application. Ensure that the application stays running by modifying the following settings: 1. Enable the optional Application Initialization feature. In Windows Server 2016, open **Server Manager** and click **Dashboard > Quick Start > Add Roles and Features**. The **Add Roles and Features** wizard opens. In the Server Roles step, click **Web Server (IIS) > Web Server > Application Development > Application Initialization**. In Windows 10, open the **Control Panel** and click **Programs and Features > Turn Windows Features On or Off**. Click **Internet Information Services > World Wide Web Services > Application Development Features > Application Initialization**. 2. In the IIS Manager, click **Application Pools** in the **Connections** panel. 3. In the workspace, right-click the application pool and click **Advanced Settings**. 4. In the General settings, set **Start Mode** to **AlwaysRunning**. 5. In the Process Model settings, set **Idle Timeout** to **0**. 6. In the CPU section, set **LimitInterval** to **0**. 7. In the Recycling section, set **Regular Time Interval** to **0**. 8. In the Generate Recycle Event Log Entry node, under the Recycling section, set **Regular Time Interval** to **False**. Click the **OK** button. 9. Back in IIS Manager, right-click on your website and click **Manage Website > Advanced Settings**. 10. In the General settings, set **Preload Enabled** to **True**, then click **OK**. This ensures that the application starts up immediately after a recycle. #### Configure ASP.NET App Pool Recycling IIS recycles regularly so that it can clean up the ASP.NET app pool processes. You can schedule recycling during off-peak hours instead by navigating to the Recycling section, setting **Specific Time** to **True**, and entering the times to recycle in the format hh:mm:ss. #### Preloading Applications IIS's preloading feature improves performance by allowing the application to run prior to a user connecting. To enable this feature, right-click the web application associated with Arc in the **Connections** pane and select **Manage Application > Advanced Settings**. In the PreloadEnabled menu, select **True**. When PreloadEnabled is set to **True**, IIS simulates a user request to the default page of the website or virtual directory so that the application initializes. #### Confirm Settings To open the application, navigate to `http://localhost/arc`. If you get any errors, see [Troubleshooting IIS](#troubleshooting-iis). #### Host TLS/SSL Connections (IIS) You can use TLS/SSL to protect the confidentiality of your business-critical and mission-critical communications with trading partners. The following steps assume you already have a certificate you can use to enable TLS/SSL on your server. 1. In IIS Manager, click your website's node from the **Connections** pane. 2. Click **Bindings** on the **Actions** pane. 3. Click **Add** and select **HTTPS**. 4. Choose the server certificate. 5. If you want to require TLS/SSL, double-click the SSL/TLS Settings icon in the workspace with your website's node still selected. Select **Require SSL/TLS** and click **Apply** in the Actions pane. ### Windows Authentication Configuring Arc to use Windows Authentication when Arc is hosted in IIS is similar to the configuration required to use the .NET [embedded web server](#using-the-embedded-server). However, instead of making changes inside the embedded web server, you make the necessary changes inside IIS. To set up Windows Authentication when Arc is hosted in IIS, follow these steps: 1. Log into Arc and add all Windows AD [Users](../administration/settings/users) who need access to Arc to the application. Windows AD Arc user names must use the following syntax: *DOMAIN\Username*. > [!NOTE] > > You must add users individually, because an entire group cannot be granted access to the application. 2. Open IIS to stop the site that is configured for Arc. Right-click on the site and select **Manage Website > Stop**. 3. Click the Arc site to open the **Home** page. Select **Authentication** under the **IIS** section. ![IIS authentication settings page](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/iis_authentication.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=eddd5eafb603adaf625b246b02b4a4f0) A screen appears listing a set of authentication methods. Right-click **Forms Authentication** and select **Disable**. This turns off the default authentication method that requires logging in with a username and password. 4. Right-click **Windows Authentication** and select **Enable**. 5. Restart your site in IIS. You might need to edit your Windows Authentication providers in IIS so IIS no longer prompts you for a username and password when accessing Arc. To do this from the IIS authentication settings, right-click **Windows Authentication** and select **Providers**. A list of providers opens: ![IIS authentication providers list](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/iis_providers.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=2fab73530eb908bbd2de6b844684d136) Ensure that **NTLM** is at the top of the list so that provider has priority when Arc is authenticating. You might need to completely remove **Negotiate** to stop the username and password requests. Once Windows Authentication has been correctly configured in IIS, when you access the application you are automatically logged into Arc as the logged in Windows AD user. ### Troubleshooting IIS The sections below provide resolutions for several common errors. #### The application stops responding some time after I log out IIS might be unloading your application after it times out from inactivity. By default, IIS terminates the worker process assigned to the application after 20 minutes of inactivity. See [Prevent Application Processes from Unloading](#prevent-application-processes-from-unloading) to learn how to override this feature. #### Visiting [http://localhost/application](http://localhost/application) results in a "Page Cannot Be Displayed" error (an HTTP 404 error) This can be an indication that ASP.NET is not enabled. If you are able to browse to a static image like the favicon.ico successfully (for example, `http://localhost/arc/favicon.ico`), but not to login.rst, this is a strong indication that ASP.NET is not enabled. In IIS, select **Control Panel > Programs (or Programs and Features) > Turn Windows Features On or Off > Internet Information Services > World Wide Web Services > Application Development Features**. Select the check box next to **ASP.NET**. #### The section is locked at a parent level Locking is either by default (`overrideModeDefault="Deny"`), or set explicitly by a location tag with `overrideMode="Deny"` or the legacy `allowOverride="false"`. This error can occur if ASP.NET is not installed. #### HTTP 503 Error: The service is unavailable This error can appear if you select a port in IIS that you have previously used with the Arc embedded web server. It means the service did not start successfully. IIS does not report any issues when starting the service, but the error appears when you attempt to reach the site. This happens because the embedded web server reserves any ports that you bind against so that any other process that attempts to bind on that port cannot make that reservation. To confirm the issue is caused by a port reservation, open a command prompt and run the following command: ``` netsh http show urlacl ``` This lists all reserved entries on the machine. If the URL that you are trying to bind is in the list (for example, `Reserved URL : http://+:8001/`) you can use another `netsh` command to remove the reservation (use the same value from the `show` list). For example, the following port removes the reservation from port `8001`: ``` netsh http delete urlacl http://+:8001/ ``` ## Managing Users When you first launch Arc, it prompts you to create a user with username/password credentials. Once the first user is created, users can be added, deleted, and managed on the **Users** tab of the [Settings page](../administration/settings/settings-page). See [Users](../administration/settings/users) and [User Roles](../administration/settings/user-roles) to learn more about user management and roles. ## Configuring the Application Directory The Arc application directory contains all of the data used by the application: configuration data, application data, logging data, certificates, etc. The application directory defaults to the following location: ``` C:\ProgramData\CData\Arc ``` You can change the application directory to a different folder, which is useful in several scenarios: * [Clustering](../administration/clustering) multiple instances of Arc * Using a shared network drive for application data * Embedding Arc within other systems accessing the same folders Changing the application directory moves the application's data files, but it does not move other application resources, such as .exe's or .jar's. ### Change the Application Directory To set a custom application directory, open the `Web.Config` file in the `www` folder of the installation directory. At the bottom of the file, inside the `` tags, locate this value: ```xml theme={null} ``` Replace the text inside the quotation marks after `value` with the new application directory. This can be any local or network path to which the application has read and write permissions. ## Configuring the Message Directory By default, Arc stores message data (the files and payloads that flow through connector Send, Receive, Sent, and Pending folders) under the application directory. You can configure a separate message directory to manage message file storage independently from configuration data. Common reasons to configure a separate message directory include: * Backing up configuration data independently of high-volume message files * Placing message storage on a dedicated volume with separate capacity or performance characteristics * Satisfying compliance or security requirements about where processed data may reside Configuration data (such as connector settings, profiles, and certificates) always remains in the application directory regardless of this setting. > [!NOTE] > > Changing the message directory on an existing installation does not move previously-written message files. Only new messages are stored under the new directory going forward; existing message data stays in its original location under the application directory. ### Set the Message Directory To configure a separate message directory, open the `Web.Config` file in the `www` folder of the installation directory. Inside the `` tags, add the following value: ```xml theme={null} ``` Replace the example path with any local or network path to which your application has read and write permissions. Arc creates the directory automatically if it does not exist. Arc restarts automatically when you save `Web.Config`, and the new message directory takes effect at that restart. If you host Arc in IIS, you can also recycle the application pool, but it is not required. If this setting is not present, all data is stored in the application directory. ## Configuring the Application Database The Arc application database stores several tables of application data, including: * Transaction Log (metadata for each transaction processed by the application) * Application Log (application-level errors and events) * Access Log (requests to the application's web endpoint) * Audit Log (user-made changes to the Arc configuration) By default, Arc uses a SQLite database in the installation directory as the application database. This database is recommended for up to 100,000 transactions. Once you hit that number, CData recommends moving to an external database. You can configure the application to use an enterprise database like SQL Server, PostgreSQL, or MySQL. Arc monitors the number of transactions in your database and alerts you when the transaction count exceeds the threshold for your database type. When you are alerted (by a banner at the top of each page in the application), you must either use the [Cleanup Settings](../administration/settings/cleanup-settings) on the **Settings > Advanced** page to reduce the number of transactions, or [move to a different database](../administration/settings/app-db-migration) with a higher transaction threshold. Use the following table to determine which database is right for your implementation. | Database | Transaction Threshold | | - | - | | SQLite | 100,000 | | MySQL | 500,000 | | PostgreSQL | None | | SQL Server | None | > [!NOTE] > > For security reasons, if you ever switch to a different application database, you must run the [integrityResetTampering](../../scripting/operations/op-integrity-reset-tampering) operation to reset the hash chain. The application database is configured in the `Web.Config` file in the `www` folder of the installation directory. It contains a commented-out `AppDb` block that describes the following `connectionStrings` tag. Set the connection string and provider name for the target database in the `AppDb` key. ### Set the Connection Strings The following sections provide example connection strings for various server configurations in the `Web.Config` file. You can set these values in plain text, or you can [generate an encrypted connection string](#generating-an-encrypted-database-connection-string) to use for the `connectionString` value. #### SQL Server ```xml theme={null} ``` > [!NOTE] > > To reduce the possibility of deadlocks when using SQL Server as your application database, CData recommends that you ensure that READ\_COMMITTED\_SNAPSHOT is enabled. #### MySQL ```xml theme={null} ``` #### PostgreSQL ```xml theme={null} ``` ### Generating an Encrypted Database Connection String Arc provides the ability to generate an encrypted connection string for your application database connection. You can use this encrypted connection string to specify the application database without storing your login credentials in plain text in the Arc configuration files. To generate an encrypted connection string, issue the following command in the installation directory where `CData.Arc.exe` is located, substituting your connection information for the example string at the end: ``` CData.Arc.exe -EncryptConnectionString server=serverName;database=databaseName;user=userName;password=passwordSample; ``` After you issue the command, the command window prints the encrypted string. For example: ``` ENCRYPTEDA:XuUY73BQHKdzWn52z6AqLvZ7uG6jlwkcsQL8yGM8sQ2Znfm1HwG6BpH+LP92mGBBCQX2/IWMal2V5a1AujfltOwResXjAijDzxJ7JmvgUt4= ``` You can then use this encrypted string in place of the plain text value for `connectionString` as shown in [Set the Connection Strings](#set-the-connection-strings). ### Specifying the Default Character Set (MySQL only) For MySQL versions such as MySQL 8.0 and later, the default character set (charset) for the database and its tables is UTF8 (specifically `utf8mb4`). However, in versions of MySQL prior to 8.0, the default charset is typically Latin1, which can cause issues with data that includes characters outside the basic Latin alphabet. If you use an older version of MySQL you can avoid these problems by configuring the database to use `utf8mb4` instead. You have two ways to make this change: update the existing database directly, or use MySQL's export and import tools to migrate the data to a new UTF8-encoded database. #### Update the Database Directly 1. Backup the database: `mysqldump -u root -p --default-character-set=latin1 --databases [database name] > backup.sql` 2. Change the default charset to UTF8: `ALTER DATABASE [database name] CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` 3. Generate SQL statements to change the default charset to UTF8 for all database tables: ``` SELECT CONCAT('ALTER TABLE ', table_name, ' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS alter_sql FROM information_schema.TABLES WHERE table_schema = '[database name]' AND table_type = 'BASE TABLE'; ``` 4. Execute the SQL statements generated in the previous step. #### Use the Export and Import Tools 1. Export the Data Definition Language (DDL) and data: ``` mysqldump -u root -p --default-character-set=latin1 --no-create-info [database name] > data.sql mysqldump -u root -p --no-data [database name] > schemas.sql ``` 2. Change the default charset in schemas.sql from `CHARSET=latin1` to `CHARSET=utf8mb4`. 3. Create a new database with UTF8 as the default charset: `CREATE DATABASE [new database name] CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` 4. Import to the new database: ``` mysql -u root -p [new database name] < schema.sql mysql -u root -p [new database name] < data.sql ``` ## Setting Login Lockouts In order to prevent brute force attacks, Arc automatically locks out users who enter incorrect passwords too many times. By default, a user who enters 6 incorrect passwords within 5 minutes is locked out for 30 minutes when the 7th try fails. You can modify these settings by editing the `Web.Config` file in the `www` folder of the installation directory. There are 3 settings relevant to lockouts: * **LockoutFailedAttempts**: the number of incorrect passwords that trigger a lockout (set this to 0 to disable lockouts) * **LockoutMinutes**: the duration of the lockout (default 30 minutes) * **LockoutTimeCheckPeriod**: the period after which the number of failed attempts is reset to 0 (default 5 minutes) Each of these can be set under the `` tag in the `Web.Config` file. For example, the following snippet disables lockouts: ```xml theme={null} ``` # Quick-Start Guide Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/quick-start-guide This page provides an overview of CData Arc. ## Installing Arc Follow these steps to install and start Arc:
  1. Download the installation file from the [Arc website](https://arc.cdata.com/trial/).
  2. Install the application:
    • For .NET and Java installations on Windows, run the setup.exe file and follow the installation steps.
    • For a Linux installation, extract the `CDataArc2026.tar.gz` file and place the contents in a location of your choice.
  3. Follow the instructions in [Starting the Server](./installation/starting-server) to start Arc.
  4. Activate your 30-day trial or enter your purchased license information.
## Using Arc After you install and run Arc according to your needs, see the pages below for information about using Arc: * See [Dashboard](./administration/dashboard) for information on how to configure your landing page to display metrics and other dashboard data that is important to you and your organization. * See [Designing a Flow](../flows/designing-a-flow) for instructions on creating your first flow. * [Connectors](../connectors) provides a detailed outline of how to use and configure each available connector. * [How-To Guides](../how-to) explores example flows for common setups. * See [Profiles](./administration/profiles) for information about the Profiles page and where profiles are used. * See [Reports](./administration/reports) for an overview of how to configure and run reports on your data. * See [Admin API](../admin-api) for a full list of actions and parameters that the Admin API supports. # June 2021 (21.0.7832) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2021-06 Release notes for CData Arc June 2021 (build 21.0.7832). ## Key Features * Added support for [Single Sign On](../administration/settings/single-sign-on). * [User Management and Roles](../administration/settings/users), including the ability to reset the admin password using the embedded web server. * PostgreSQL is now supported as a backend Application Database. * [Audit Logs](../administration/activity#audit-logs) * The Cleanup Options now support the ability to archive to Amazon S3. * The cleanup routine now ensures that the Application Logs truncate after 100,000 records by default. * Added a new top level [`Settings` page](../administration/settings/settings-page). * The .NET Edition's embedded web server no longer requires administrative privileges to run. Previous installations will continue to function as before, but new setups will use a [different path on disk for Application Directory by default](../installation/how-to-upgrade#windows-application-directory). ## Admin API ### Added * Added an `export` API Action for exporting arcflows. ### Changed * Renamed the `downloadFile` API Action to receiveFile. * Changed the `ConnectorId` input to the `certificates` API Resource to be `ConnectorIds` since it accepts a comma separated list of ConnectorIds. * Replaced the `importProfiles` API Action with `import`. * Deprecated the `Subfolder` input to the `files` API in favor. A `Subfolder` header on messages should be used instead. * Removed the `DeleteTransactions` input to the `archiveLogs` API Action. The `transactions` API Resource may be used with an appropriate OData filter to accomplish the same result. * Removed the `deleteLogs` API Action. A DELETE request to the `logs` API Resource may be used instead. * Removed the `deleteTransactions` API Action. A DELETE request to the `transactions` API Resource may be used instead. * Removed the `ExpirationDays` property in the `certificates` API Resource. ### Removed * Removed the `refreshTransactions` API Action. * Removed the `restartFile` API Action. ## Connectors ### Added * Added a [PostgreSQL Connector](../../connectors/postgresql/postgresql). * Added the ability to send the input file as binary data on the request in the REST Connector. * Added a `Max Failed Records` setting in the Advanced tab for Database Connectors in order to allow the connector to ignore a configurable number of insert failures without reporting an error. * The EmailReceive Connector now supports OAuth 2.0 as an authentication mechanism. ### Changed * As mentioned in the [Upgrade guide](../installation/how-to-upgrade#subfolders-in-file-based-connectors), file based connectors (like File, SFTP, etc.) now use a `Subfolder` header on messages to denote path information instead of subfolders of the Send or Receive folders. * All files in each connector's Send folder (including files in subfolders) will be consumed by the connector. If a previous system archived processed files or stored files that were not intended for transmission in subfolders of a connector's Send folder, they must be relocated or they will be processed by the connector. * The FTP and SFTP Connectors are now in the MFT category. * All connectors now support `Log Subfolder Scheme` and `Sent Folder Scheme` settings in order to prevent performance issues due to the accumulation of a large number of files or folders in a single folder. * The [`_message` special item](../../scripting/introduction-to-arcscript#message-_message) is now available in the After Receive event. * File content that is accessed from the After Receive event in connectors is now loaded from a temporary location on disk, and files will still be written to the Receive directory on disk even if the working copy is moved elsewhere. It is recommended to use the File Connector for cases like this going forward. * The XMLMap Connector will now enable the "Show Attributes" toggle automatically if attribute mappings are detected. * The CData Connector now supports v21 drivers instead of v20 drivers. * With the addition of the `BaseURL` setting, the `PublicDomain` setting is now deprecated. * The S3 Connector now supports connecting to other S3-like services, so the Wasabi Connector is now deprecated. * Connectors with more than one output folder now have a Receive folder on disk for each. * Removed the `OverwriteLocal` setting for file based connectors. A File Connector should be used instead. * The CData PostgreSQL driver now ships with the application instead of the PgSQL driver. * The API, CData, and Database Connectors now require a license for CData drivers in order to function. In cases where an individual connector exists for the data source, that connector may be used instead. For example, if the Salesforce driver is used in the CData Connector, it would require a license for the driver, but the Salesforce Connector would not. Please reach out to [arcsales@cdata.com](mailto:arcsales@cdata.com) to inquire about obtaining a license if necessary. ## Miscellaneous ### Added * Added an `InstanceId` column to all log tables in Application Database. This is useful in HA environments in order to determine which instance in the cluster produced the log entry. ### Changed * The Windows Edition now requires .NET 4.5 to run. * The Java Edition now requires Java 1.8 or higher. * The Java Edition now requires Servlet 3.0 to run in external java servlet containers. * Flow related configuration details are now stored in flow\.json instead of in port.cfg settings like `SendToConnector`. Arc will now migrate these flow connections from port.cfg files to flow\.json when the application starts. # January 2022 (21.0.8054) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2022-01 Release notes for CData Arc January 2022 (build 21.0.8054). ## Key Features * Health Checks are now performed hourly. These checks trigger alerts based on the size of the Application Database, memory usage, disk usage, and the number of files in single folders. * Added a [`Connections` tab in the `Settings` page](../administration/settings/connections) for connectors that are capable of using shared connections. * Added support for lockouts in the SFTPServer and OFTPServer Connectors to prevent DDoS attacks. * [Batch Groups & Batch Messages](../architecture#batch-groups-and-batch-messages) are now supported. * Added support for Object and Collection typed settings to the Admin API. These settings are now returned from GET requests to the Admin API as base64 encoded data, and can likewise be set by base64 encoding the object's content in update requests. * The Admin API now accepts ranges of IP addresses in CIDR notation. A Description field has also been added to the table to make it easier to keep notes on each entry. * Added a *copyWorkspace* action to the Admin API. ## Connectors ### Added * Added the ability to choose between using the global or connector-specific proxy settings in the AS2 Connector. * The EmailReceive Connector now supports an `Attachment Filter` setting. * Added a [Salesforce Connector](../../connectors/salesforce/salesforce). * Added a [Shopify Connector](../../connectors/shopify/shopify). * Added a `GET Request Body` toggle to the Advanced page of the REST Connector. This toggle provides the ability to use the input file as the body of a GET request. * Added the ability to [provide a custom response to requests](../../connectors/webhook#custom-responses) in the Webhook Connector. * The X12 Connector now provides complete schema examples for mapping when connected to an XML Map Connector. * The XML Map Connector now allows for searching the list of Source and Destination Files when more than 10 files are available. * Added a [Magento Connector](../../connectors/adobe-commerce/adobe-commerce). * Added a [BigCommerce Connector](../../connectors/bigcommerce/bigcommerce). * Added an [Apache Kafka Connector](../../connectors/apachekafka). * Added new versions of each connector in the Database category with a new single-action design. The older (multi-action) database connectors, as well as the Lookup Connector, are now deprecated and will be removed in a future release. * Added the ability to surface headers from the response as metadata on the output message in the REST Connector. * Added a [Zoho CRM Connector](../../connectors/zoho-crm/zoho-crm). ### Changed * Number formatters now use the `BigDecimal` type by default in the .NET Edition. Previously, `double` was used, which could lead to precision loss. To revert back to using `double`, set the `FormatterNumberType` key to `double` in the appSettings section of Web.Config. * The dbCall and dbQuery scripting operations now accept parameters through `paramname#`, `paramtype#`, and `paramvalue#` inputs instead of `inparam:*`. As a result of this change, `inparam` is no longer functional. * The Home Directory settings for GISB and OpenPGP Connectors are now deprecated. * The EmailSend Connector now uses a single connection when sending multiple messages through a single worker in the automation service. * The AS2 and RNIF Connectors now use [Batch Groups](../architecture#batch-groups-and-batch-messages) for attachments. * The `batchResults` attribute in Output Mappings for Database Connectors is now deprecated. The *Max Records* setting in the Advanced tab should be used instead. * Database connectors (e.g. MySQL, SQLite, etc.) are now in their own category and are licensed as part of the Professional tier. Existing licenses for these connectors will still be respected to preserve backward compatibility. * Connectors that allow for the specification of Trusted IP Addresses (like SFTPServer, FTPServer, etc.) now accept ranges of IP addresses in CIDR notation. A Description field has also been added to the table to make it easier to keep notes on each entry. # April 2022 (21.0.8126) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2022-04 Release notes for CData Arc April 2022 (build 21.0.8126). ## Key Features * Performance improvements for the flow designer. * Added a *setFlow* action to the Admin API. * The flow designer now automatically saves changes to the flow. * Added the ability to undo and redo changes made to the flow within the flow designer. ## Connectors ### Added * Added a [NetSuite Connector](../../connectors/netsuite/netsuite). * Added a [QuickBooks Online Connector](../../connectors/quickbooks-online/quickbooks-online). * Added a [ShipStation Connector](../../connectors/shipstation/shipstation). * Added an [Amazon Marketplace Connector](../../connectors/amazon-marketplace/amazon-marketplace). ### Changed * The [XML Map Connector](../../connectors/xml-map/xml-map) now shows lines between mapped nodes to better visualize data mapping. * The [HL7 MLLP Connector](../../connectors/hl7mllp) has moved out of Beta status into Full release. # June 2022 (22.0.8215) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2022-06 Release notes for CData Arc June 2022 (build 22.0.8215). ## Key Features * Added a new top level [Reports](../administration/reports) page. * Improved the experience for [Importing and Exporting Flows](../administration/importing-exporting-flows). Sensitive values included in Arc flows can now be secured with a custom password. * In the flow designer, each connector now displays a badge that indicates the number of messages that are pending processing. This can help to identify bottlenecks in busy flows where assigning more worker threads may lead to better overall performance. * Added support for sending application logs to a Syslog server in Settings > Alerts > Syslog Settings. * CData Arc now allows for immediate cleanup of the application database and disk. If the Settings > Cleanup Options > Interval is set to 0, cleanup tasks will clear all entries from the database and disk (a value of -1 will skip cleanup). * The Embedded Web Server that ships with the Java Edition has been updated to Jetty 10, and now enables gzip compression by default. ## Connectors ### Added * Added a [WooCommerce Connector](../../connectors/woocommerce/woocommerce). * Added a [Dynamics 365 Business Central Connector](../../connectors/dynamics-365-bc/dynamics-365-bc). * Added a [Tally Connector](../../connectors/tally/tally). * Added a new version of the [SQL Server Connector](../../connectors/sql-server/sql-server) that provides support for better authentication mechanisms. The old version of the SQL Server Connector is now deprecated and planned for removal in CData Arc 2023. * Added a new version of the [SharePoint Connector](../../connectors/sharepoint/sharepoint) that provides support for better authentication mechanisms. The old version of the SharePoint Connector is now deprecated and planned for removal in CData Arc 2023. * Added the ability to leverage Amazon STS in order to assume an IAM role by name in the [S3 Connector](../../connectors/s3). * Added a [Square Connector](../../connectors/square/square). # October 2022 (22.0.8336) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2022-10 Release notes for CData Arc October 2022 (build 22.0.8336). ## Key Features ### Added * Added the ability to expose flows as APIs through the new [Flow API](../../flows/flow-api) feature. * Added the [Global Settings Vault](../administration/settings/global-settings-vault) feature. * The CData Drivers embedded within CData Arc have now been upgraded to the V22 release. * Usage of the Oracle MySQL driver within CData Arc is now deprecated. It is recommended to migrate to the CData MySQL driver for MySQL connections in connector configurations or [AppDB](../installation/cross-platform-edition#mysql). ### Fixed * Fixed an issue preventing a password protected export of the Default workspace in some scenarios. * Fixed an issue that could cause an automation cycle to be unnecessarily triggered upon upgrading from older versions. * Fixed an issue that prevented some instances that used SQL Server as the Application Database from cleaning up records. ## Connectors ### Added * Added an [Exact Online Connector](../../connectors/exact-online/exact-online). * Added a [Sage Intacct Connector](../../connectors/sage-intacct/sage-intacct). * Added a [Square Connector](../../connectors/square/square). * Added the ability to use wildcards in the Path setting of the [File Connector](../../connectors/file). * Added a [Snowflake Connector](../../connectors/snowflake/snowflake). * Added a [Databricks Connector](../../connectors/databricks/databricks). * Added an [Odoo Connector](../../connectors/odoo/odoo). * Added a [Dynamics 365 Connector](../../connectors/dynamics-365/dynamics-365). * Added a [Batch Create Connector](../../connectors/batch-create). * Added a [Batch Split Connector](../../connectors/batch-split). * Added a [Batch Merge Connector](../../connectors/batch-merge). * Added an [AMQP Connector](../../connectors/amqp). * Added a [Stripe Connector](../../connectors/stripe/stripe). * Added a [Zoho Inventory Connector](../../connectors/zoho-inventory/zoho-inventory). * Added a [FHIR Connector](../../connectors/fhir/fhir). ### Changed * The [NetSuite Connector](../../connectors/netsuite/netsuite) has moved out of Beta status into Full release. * The [ShipStation Connector](../../connectors/shipstation/shipstation) has moved out of Beta status into Full release. * The [Amazon Marketplace Connector](../../connectors/amazon-marketplace/amazon-marketplace) has moved out of Beta status into Full release. * The [SharePoint Connector](../../connectors/sharepoint/sharepoint) has moved out of Beta status into Full release. * The [Dynamics 365 Business Central Connector](../../connectors/dynamics-365-bc/dynamics-365-bc) has moved out of Beta status into Full release. * The [Shopify Connector](../../connectors/shopify/shopify) now supports Access Token authentication, and no longer supports Basic Authentication. OAuth Authentication now requires OAuth Client Id and OAuth Client Secret to be specified. * The [Dropbox Connector](../../connectors/dropbox) is now able to refresh access tokens automatically. * Some legacy versions of the application allowed the root directories for FTPServer and SFTPServer to overlap with the connector's folder within Application Directory where configuration data was stored. This configuration is no longer supported. If the root directory for FTPServer or SFTPServer in your instance meets this criteria, it is recommended to relocate the root directory to be a new folder at the root of Application Directory (e.g. "\/sftpserver"). ### Fixed * Resolved a thread conflict issue in the SFTP Server Connector when under heavy load. * Fixed an issue that caused overriden Local Folders in SFTP Server to behave unexpectedly after upgrading. * Fixed an issue where some connectors within workspaces that contained special characters in the Workspace Id were unable to use the state database. * Fixed issues in the SharePoint Connector (while in Beta) related to uploading to subfolders and creating those folders if they did not already exist. * Optimized memory usage when the AS4 Connector received requests that did not match a trading partner profile. * Fixed an issue that caused FinalAttempt to be set to False in the afterSend event even though MaxAttempts had already been exhausted for that connector. ### Removed * The Payment and ACH Connectors have been removed. # January 2023 (22.0.8473) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2023-01 Release notes for CData Arc January 2023 (build 22.0.8473). ## Key Features ### Added * The [Import and Export](../administration/importing-exporting-flows) experience has been rebuilt from the ground up, adding the ability to perform global imports and exports, selective imports, conflict resolution, and more. * The CData SQL Driver can now be utilized for Application Database connections that utilize a SQL Server database. As a result, connectivity to each backend database type that is supported is possible without installing any additional drivers. * Added the ability to route a copy of successfully processed input messages down an alternate path in the flow through a new "Show Success Path" context menu option on connector shapes in the flow designer. * [Reports](../administration/reports) now include a Since Last Run option for the Time Period. * Added the ability to generate certificates with a keys size of 4096. * Added the ability to import the sample flows from [arc.cdata.com/flows/](https://arc.cdata.com/flows/) directly in the flow designer. ### Changed * The .NET Edition now runs as a Windows Service by default for new installations. * Significant performance improvements across the whole UI, especially in environments that leverage a networked file system. * Improved the performance of loading large Batch Groups in the UI. * Improved the performance of tracked headers in the Application Database. * Added a None option to the Auth Scheme option in Settings → Alerts. * The Alerts tab in the Advanced page now allows the user to opt in to using the global proxy settings. * The cleanup routine now obeys the Cleanup Type specified for the cleanup of Sent and Send folders in addition to the Logs folder. ### Fixed * Fixed an issue in the Java Embedded Web Server that prevented the server from starting in some rare situations. This was due to an externally hosted DTD that started redirecting to a new location. As a result of this change, the embedded web server prefers local references to external ones. * Fixed a rare issue retrieving profile settings for some services in environments with very high concurrency. * Improved the memory utilization for the [FTP Server Connector](../../connectors/ftp-server). * Fixed various minor UI issues in Safari. * Fixed an issue in the [Salesforce Connector](../../connectors/salesforce/salesforce) where OAuth tokens would not be refreshed in some scenarios. * The SQLite dll that ships with the .NET Edition has been updated to address CVE-2022-35737. * Fixed an issue in the [sysExecute](../../scripting/operations/op-sys-execute) op where unexpected inputs were causing errors. * Fixed an issue with [SSO](../administration/settings/single-sign-on) when the web interface is accessed through a proxy. * Fixed an issue with the Weekly Status Report that caused inaccurate results in some rare situations. ## Connectors ### Added * Added [Submit Vendor Shipment Confirmations](../../connectors/amazon-marketplace/amazon-marketplace-submit-vendor-shipment-confirmations), [Submit Vendor Order Acknowledgement](../../connectors/amazon-marketplace/amazon-marketplace-submit-vendor-order-acknowledgement), and [Submit Vendor Payments](../../connectors/amazon-marketplace/amazon-marketplace-submit-vendor-payments) actions to the [Amazon Marketplace Connector](../../connectors/amazon-marketplace/amazon-marketplace). ### Changed * Errors that arise from a passive receive now trigger an alert. * Attempts to manually receive files in MFT Connectors where no files were found to download now allow for the download of logs in order to understand why no files were downloaded. * The [Notify Connector](../../connectors/notify) now allows for the evaluation of expressions in the message that it sends. * Database type connectors (Database, CData, Salesforce, etc.) now allow for their state data to be reset from the UI in order to be able to reprocess rows of data that may have already been processed in a previous execution attempt. * Connections within the Settings → Connections tab of Arc now allow for the retrieval of logs associated with Connect and Test Connection attempts. * More examples have been added to various scripting contexts as new snippets. * All connectors now output using [Arc's Message format](../architecture#message-implementation) by default. Previously, a connector that was not connected to another connector in the flow would write output to the Receive folder in raw format. If this behavior is still desired, a File Connector should be used instead. * The Add Table modal in Database type connectors now supports the ability to search through the available tables. * The CustomerAddress column in the Invoices table for the [Stripe Connector](../../connectors/stripe/stripe) is now translated as an XML aggregate to allow for a more seamless mapping experience. * The [Square Connector](../../connectors/square/square) has moved out of Beta into Full release. * The [Tally Connector](../../connectors/tally/tally) has moved out of Beta into Full release. * The [WooCommerce Connector](../../connectors/woocommerce/woocommerce) has moved out of Beta into Full release. * The DataRobot Connector is now deprecated and planned for removal in CData Arc 2023. ### Fixed * Message headers added through the \_message item in the [Script Connector](../../connectors/script) are now appropriately reflected on output messages. * Fixed an issue in the [SFTP Server Connector](../../connectors/sftp-server) that caused some client connections to be dropped when multiple authentication methods are attempted in the same connection. * The [Dynamics 365 Connector](../../connectors/dynamics-365/dynamics-365) now shows the OAuth Client Id and OAuth Client Secret fields in the Connection modal in order to utilize grant types for which no OAuth information is embedded. * Fixed an issue in the [Zoho Inventory Connector](../../connectors/zoho-inventory/zoho-inventory) that caused an Invalid URI to be produced when testing the connection. * Fixed an issue that resulted in [AS4 Connector](../../connectors/as4) configurations carried over from previous versions not being compatible with the current version. # April 2023 (23.0.8517) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2023-04 Release notes for CData Arc April 2023 (build 23.0.8517). ## Key Features ### Added * Added a new [Dashboard](../administration/dashboard) page to provide an overview of activity when logging into the application, assess trends, and identify problems that might need attention. As part of this change the Status page has been removed, and the logs that were previously found there have been moved to their own dedicated [Activity](../administration/activity) page. * Added the ability to view the entire lifecycle of a message as it travels through the flow in one place. More details can be found [here](../administration/activity#messages-tab). * Added a new [Searching and Filtering](../administration/activity#searching-and-filtering-the-logs) experience for all tables, including designer-driven support for filtering by tracked headers on messages. Input and Output tabs for connectors now check for new data to show in the table in the background, and show an indicator if more data is available to be shown. ### Changed * The Scheduler tab for [Reports](../administration/reports) now shows the time zone in which the schedule will run. * Updated the Embedded Jetty Web Server to 10.0.10 to address CVE-2022-2048. * The Add User and Edit User modal in the Users tab within the Settings page now allows rate limiting options to be set for users who have access to the Admin API. * The [todate](../../scripting/value-formatters/date-formatters#todate-outputformat-inputformat-strictinputformat) formatter now allows for a third strictInputFormat input to force the formatter to fail if the specified input format does not match the input exactly. * Relaxed the validation of the Content-Type header for the [Flow API](../../flows/flow-api) to allow for various other attributes to be present. * Updated the SQLite library in the .NET edition to more closely match the expected .NET runtime for the application. As part of this change, the .NET edition now requires Visual C++ 2013 Update 2 and packages that version within the setup. ### Fixed * Fixed an issue migrating early versions of caching metadata to the state database for connectors like S3, SFTP, and FTP. * Fixed various locking issues that could potentially be seen under very high concurrency. * Updated the build of the SQL driver to address various stability issues when using SQL Server as AppDB. * Improved the performance of the flow designer when the list of sample flows could not be obtained, or took a very long time to be returned due to poor network conditions. * Fixed an issue causing query parameters to not be added as message headers for certain body types in the [Flow API](../../flows/flow-api). * Fixed a rare issue where certain sequences of characters in headers could cause display issues. * Fixed a medium level security vulnerability in the .NET Embedded Web Server. ## Connectors ### Added * [SFTP Server](../../connectors/sftp-server) connectors now add an SFTPServer-Virtual-Path header to received messages so the path at which those files are received by the connector can be referenced later in the flow. * The [SOAP](../../connectors/soap) connector now allows for KeyIdentifier references for the Security Token Format. ### Changed * The validation of incoming requests for the [AS4](../../connectors/as4) connector was improved to also include the Service and Action on the request. * Various HIPAA schemas were added to the list of available values in the GS08 setting in the [X12](../../connectors/x12) connector. * New instances of [Zip](../../connectors/zip) connectors now compress items directly to the root of the archive when operating in Scheduled Send mode. Existing instances of Zip connectors continue to use a static "Send" folder within the archive to place these files. * Improved the [IBM MQ](../../connectors/ibmmq) connector's JMS support to properly handle binary messages. * The following connectors have moved out of beta into full release: * [AMQP](../../connectors/amqp) * [Databricks](../../connectors/databricks/databricks) * [Dynamics 365](../../connectors/dynamics-365/dynamics-365) * [Exact Online](../../connectors/exact-online/exact-online) * [FHIR](../../connectors/fhir/fhir) * [Odoo](../../connectors/odoo/odoo) * [Sage Intacct](../../connectors/sage-intacct/sage-intacct) * [Snowflake](../../connectors/snowflake/snowflake) * [Stripe](../../connectors/stripe/stripe) * [Zoho Inventory](../../connectors/zoho-inventory/zoho-inventory) * [EDI](../../connectors/edi-connectors) connectors now support EDI-InterchangeReceiverIdQualifier and EDI-InterchangeSenderIdQualifier headers to specify their respective values on acknowledgements. * Updated all CData drivers that are bundled with the application. ### Fixed * Improved batch processing for database connectors to properly include NULL values as parameters instead of directly in the query. * Updated the SharePoint driver to address issues receiving files in the [SharePoint](../../connectors/sharepoint/sharepoint) connector for certain environments. * Fixed an issue that caused various connectors to be unable to retrieve OAuth tokens within workspaces that contained reserved characters. * Fixed an issue in the [Email Receive](../../connectors/email-receive) connector that prevented the retrieval of emails that contained reserved characters in the to, from, and cc fields. * Fixed an issue preventing Public Key authentication in the [SCP](../../connectors/scp) connector to work for some server configurations. * Addressed an inconsistency in the response that the [API](../../connectors/api/api) connector served in Java for calls that targeted a resource by Id. * The following connectors have been removed from the application: * DataRobot * Wasabi # July 2023 (23.2.8601) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2023-07 Release notes for CData Arc July 2023 (build 23.2.8601). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/VRVcNzR9YHM](https://www.youtube.com/embed/VRVcNzR9YHM) ## Key Features ### Added * Added the ability to filter by ConnectorId in the [Messages](../administration/activity#messages-tab) tab. * Added the ability to download the logs for various UI interactions, like the Send Test Email button in Settings → Alerts, and Connect buttons for OAuth. * Added a button in the preview modal for messages and logs to copy contents to the clipboard. * Added support for EC keys to [SSO](../administration/settings/single-sign-on). * Added a QueryPassthrough input to the [dbQuery](../../scripting/operations/op-db-query), [dbNonQuery](../../scripting/operations/op-db-non-query), and [dbCall](../../scripting/operations/op-db-call) operations. ### Changed * Relaxed the locking requirements for [Flow APIs](../../flows/flow-api) to allow for better performance and parallelism. * Health check alerts for memory usage now mention the InstanceId of the node that triggered the alert to make it easier to diagnose problems in multi-node environments. * Increased the password complexity requirements for new passwords. ### Fixed * The Windows edition setup now appropriately detects if Application Directory is within Installation Directory and skip the process of modifying the permissions of that folder. * Alerts are now triggered for certificate and license expirations regardless of the selection for error notifications. * Fixed some minor UI issues in the flow designer. ## Connectors ### Added * Many improvements to the [XML Map connector](../../connectors/xml-map/xml-map), including designer driven support for variables, more flexible conditions, notifications, real-time validation for ArcScript, and the ability to test the mapping directly in the Settings tab. * Added the ability to define multiple criteria for a source value in the [Branch connector](../../connectors/branch), and create a separate path in the flow for each criteria. * Added the ability to define more than two output paths for a [Copy connector](../../connectors/copy). * Added an [XSD connector](../../connectors/xsd). * Added the ability to discard the response body in the [REST connector](../../connectors/rest). The Output Behavior setting provides options for None, Request, and Response. * The [Zip connector](../../connectors/zip) now supports creating an archive from all the Batch Messages in a Batch Group. * HIPAA schemas are now included with the [X12 connector](../../connectors/x12). ### Changed * The [BatchMerge connector](../../connectors/batch-merge) now uses a better default for the Local File Scheme setting that ensures that output files are written a with an XML extension. * Performance improvements in many connectors. * The [CSV](../../connectors/csv), [TSV](../../connectors/tsv), and [PSV](../../connectors/psv) connectors now allow for root XML elements with any name. Previously, "Items" was required to be the root element name. ### Fixed * Fixed an issue causing Execute Stored Procedure actions to fail in the [PostgreSQL connector](../../connectors/postgresql/postgresql) for some stored procedures. * Fixed an issue preventing the [SharePoint connector](../../connectors/sharepoint/sharepoint) from downloading files with long names in the Windows edition. * The [SharePoint connector](../../connectors/sharepoint/sharepoint) now appropriately detects new document libraries that are created after a connection is already established. * Fixed an issue causing files that failed to receive a successful asynchronous acknowledgement to be resent more than the configured number of attempts. * Fixed an issue in the [FlatFile connector](../../connectors/flat-file) that prevented proper detection of the control field in complex multi-line position delimited files. * Fixed an issue preventing some files in the Additional Paths within the [SFTP Server connector](../../connectors/sftp-server) from being served to clients in some rare situations. * Fixed an issue preventing the [FTP](../../connectors/ftp) and [SFTP](../../connectors/sftp) client connectors from downloading very large files in some situations. * Connectors that handle incoming HTTP requests now wait until the application has fully started before serving requests. Previously, there was a small window where the connector could attempt to serve the request and result in an error due to some resources not being fully initialized. * The [X12 connector](../../connectors/x12) no longer uses the `Repetition Char` setting ('^' by default) for documents using versions before 00404. # October 2023 (23.3.8698) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2023-10 Release notes for CData Arc October 2023 (build 23.3.8698). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/2M0DZTwYh28](https://www.youtube.com/embed/2M0DZTwYh28) ## Key Features ### Added * The [Java Edition](../installation/cross-platform-edition) now includes a guided setup experience for Windows, with the capability to run as a service. For Linux installations, a .tar.gz file is now available. Both options now include a JRE so an external installation is no longer necessary. * Added the ability to format message content in the preview modal for XML and JSON files. * Added the ability to filter the [dashboard](../administration/dashboard) for transactions from the current day. Trends are assessed by comparing metrics to the same time period from the previous day. * Metric cards on the [dashboard](../administration/dashboard) now include links to more detailed information, where applicable. * [SSO](../administration/settings/single-sign-on) issuer certificates are now refreshed daily, or upon failure to verify the signature in a JWT. ### Changed * All CData drivers included in the application have been updated to 2023. * Alerts for errors that pertain to a connector now mention the Id of the workspace that the connector resides in. * [Flow APIs](../../flows/flow-api) now do not create locks by default to allow for greater concurrency. ### Fixed * Fixed an issue where database connectors receiving a large number of records in a single receive operation could result in an Out of Memory exception in some environments. The Arc messaging engine now appropriately buffers message data to disk when a large number of messages are produced in a single receive operation. * Fixed an issue causing the process of populating message-related metadata from the transactions table to fail if SQL Server was used as Application Database. This only affected the working set of data present at the time of upgrading. * Fixed an issue causing health checks to fail to report the disk usage of Application Directory when set to a UNC path. If Archive Directory is set to a separate disk, it is now checked as well. * When copying one or more connectors into the same workspace as the original connectors, the copies are now placed directly beneath the original connectors. * Minor security fixes to address low level vulnerabilities. ## Connectors ### Added * Added a [Kintone](../../connectors/kintone/kintone) connector. * Rebuilt the lookup experience in database connectors to provide several key improvements: * The Lookup action now allows for selecting multiple columns and records within the same operation. * Added a designer to the Lookup action to assist in building queries, referencing information from input files, and testing the configuration directly in the settings tab. * Added the ability to target a stored procedure with a lookup through the Lookup Stored Procedure action. The existing Execute Stored Procedure action has been improved to match this design. ### Changed * When a custom script or condition in the [XML Map connector](../../connectors/xml-map/xml-map) causes the connector to throw an error, the error now references that node by name. * [XML Map](../../connectors/xml-map/xml-map) no longer hides namespace definitions in the mapping editor. * Improved the performance of services that receive files passively, especially for connectors like [AS2](../../connectors/as2) when requests do not match a configured trading partner profile. * The [Validate connector](../../connectors/validate) now allows for linked schemas to pass through it. For example, in a Database → Validate → XML Map flow, the XML Map connector can now reference the schema from the Database connector for mapping. * Improved the performance of the [PDF Label connector](../../connectors/pdf-label). * The [FTP Server connector](../../connectors/ftp-server) now requires the reuse of the PI SSL session in the DI channel by default. * The Magento connector has been rebranded to the [Adobe Commerce connector](../../connectors/adobe-commerce/adobe-commerce). * The [Excel connector](../../connectors/excel) now allows for the specification of a range of cells where the table should be found in Table mode. ### Fixed * The [SharePoint connector](../../connectors/sharepoint/sharepoint) now better handles long filenames. * Fixed an issue in the UI of the [SFTP connector](../../connectors/sftp) that caused the server thumbprint to be overwritten when creating a new public key. * Application connectors like [Salesforce](../../connectors/salesforce/salesforce), [Shopify](../../connectors/shopify/shopify), etc. now properly handle aggregate columns when a custom query is defined. * The [Email Receive connector](../../connectors/email-receive) now properly handles message attachments that are not part of MIME encoded entities. # January 2024 (23.4.8839) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2024-01 Release notes for CData Arc January 2024 (build 23.4.8839). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/UeFk1UnD9kI](https://www.youtube.com/embed/UeFk1UnD9kI) ## Key Features ### Added * Added the ability to create custom roles with up to 10 policies, allowing users access to only certain actions within specific workspaces. For more details, see the [Custom Roles](../administration/settings/user-roles#custom-roles) section. * Added the ability to specify expected volumes for connectors. If these volumes are not met within the specified time frame, users are made aware via alerts, dashboard widgets, and a dedicated tab in the Activity page (previously Logs). For more details, see the [Service Level Agreements (SLAs)](../administration/activity#slas-tab) section. * UI improvements for the flow designer, including but not limited to, the ability to delete connectors within the settings pane, showing the connector type in the shape for a connector on the canvas, as well as the description for a connector on hover. * A new "Include files in Receive folders" setting has been added to Settings → Advanced → Cleanup Options. Complete flows usually include terminal connectors that consume files, ensuring they do not accumulate at the end of flows. In flows where this is not the case, files would accumulate in the Receive folders and could potentially cause performance issues. Enabling this setting ensures that these files are cleaned up. Note that as part of this change, the directory structure of the archive changed to keep logs separate from message data. * Added a new [xpathcount](../../scripting/value-formatters/function-formatters) formatter. This can be used within xmlDOMSearch operation calls, or in scripts within XML Map in order to easily determine the number of elements in the document that match the given xpath. ### Changed * Relaxed locking requirements for the [Admin API](../../admin-api) to improve performance and concurrency. * Versions of MySQL prior to 8 are now deprecated as an Application Database. * Improvements to performance and stability for Derby and SQLite databases when they are used as an Application Database. * The [dbQuery](../../scripting/operations/op-db-query) op previously produced an output attribute for every column returned from the query, even when the value for that column's value was null. In order to facilitate detection of null columns, they are now excluded on the output item produced by the op. Scripts that utilize dbQuery to obtain column values that might be null should be updated to include the def formatter to ensure that errors do not arise from the absence of these attributes on the output item. * Some [scripting operations](../../scripting/operations/operations), specifically connector operations like [x12Scan](../../connectors/x12#x12-operations), must be called through the `connector.rsc` endpoint with an auth token. The ability to call these operations directly is now deprecated and will be removed in a future release. ## Connectors ### Added * Connectors that interact with remote file systems (SFTP, File, S3, etc.) now preserve the full remote path by adding a `RemotePath` header to messages received by the connector. * The [Email Send](../../connectors/email-send) connector now supports sending multiple attachments in one email through the use of Batch Groups. * Email addresses for alerts can now be overridden at a connector level in the Alerts tab. * Connectors that use OAuth for authentication now show a warning when the tokens that were retrieved are unable to be refreshed. * Added the ability to format EDI data when previewing these messages within the UI. * The [REST](../../connectors/rest) connector now allows for form fields on requests to be populated from message headers and ArcScript. As part of this change, the existing "Dynamic" option for form fields was renamed to "XML", and new "Header" and "ArcScript" options were added. * The [Email Send](../../connectors/email-send) connector now allows for the Content-Type of attachments to be determined by a new "Extension Map" setting, as well as a Content-Type header on input messages. ### Changed * The [AS2](../../connectors/as2) connector now positively acknowledges an incoming request before the full request body is read when an asynchronous MDN is requested. * Improved the [X12](../../connectors/x12) connector's exception messages and support for the Upload Test File feature, particularly the way it interacts with other features of the connector, like Nest Master-Detail Loops. * [EDI](../../connectors/edi-connectors) connectors now generate an alert when a negative acknowledgement is received. * Connectors that translate flat file formats (like CSV, TSV, and PSV) now preserve original column names in a `flatfile:column` attribute on the translated XML when it differs from the translated version. These connectors also read from this attribute when translating from XML to flat file formats in order to determine the header name. * The [Kintone](../../connectors/kintone/kintone) connector moved from beta to full release status. ### Fixed * Fixed several minor issues in the [Flat File](../../connectors/flat-file) connector. * Improved the [Validate](../../connectors/validate) connector to better handle validations of messages that contain many repeated elements. * Fixed an issue in the [Split](../../connectors/split) connector where significant whitespace was not preserved in output XML in some situations. * Fixed the IsNull operator in the [Branch](../../connectors/branch) connector to better handle missing headers and XML elements that are explicitly set to null with the `xsi:nil` attribute. * The [FTP Server](../../connectors/ftp-server) and [SFTP Server](../../connectors/sftp-server) connectors now utilize the `Subfolder` header on messages to place messages from the flow in their appropriate directories within the server's root directory. * Performance and stability improvements in the [SFTP Server](../../connectors/sftp-server) connector, especially when Temp File Extensions are used, and clients drop connections unexpectedly. # April 2024 (24.1.8949) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2024-04 Release notes for CData Arc April 2024 (build 24.1.8949). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/qoIApgJCZes](https://www.youtube.com/embed/qoIApgJCZes) ## Key Features ### Added * Added a [Workspaces Grid](../../flows/workspaces#using-the-workspaces-grid) page that includes a summary of all the workspaces a user has access to. Features include searching, filtering, and a visualization of the flow. * Rebuilt the [flow designer](../../flows/designing-a-flow) from the ground up. The redesign includes many new features, including, but not limited to: * Better visualization for the beginning and end connectors in the flow through new Start Trigger and Flow End shapes. As part of this change, connectors and any available actions in those connectors are now categorized and displayed according to whether they are considered to be a [Trigger, Transform, or Terminal](../../connectors#categories). * Real-time updating context for connector shapes in the flow to make it easier to see what a connector's role in the flow is. These real-time updates include SLA statuses, transaction counts, warnings, and more. * Better visualization of connectors that are capable of performing multiple actions, like AS2. These actions now show as distinct shapes in the flow designer, with links to navigate between each action, and the ability to hide an action if it is not needed. * A new [Add Connector](../../flows/designing-a-flow#explaining-the-add-connector-modal) experience that shows the available actions up front, along with a preview of what the connector would look like, as well as tooltips that explain each UI element in the connector's shape. * The `_message` [item in ArcScript](../../scripting/introduction-to-arcscript#logging-connector-events) now supports logging through a new log attribute prefix. When combined with a suffix that matches a log level, the value of the attribute is added to the logs for the message (for example, ``) * The `xpath` formatter (and features that make use of it, like the [XML Map connector](../../connectors/xml-map/xml-map)) now supports the `name()` function. ### Changed * The legacy portOps scripting operations must now be called through their respective Admin API endpoints using `api.rsc` as described [here](../../admin-api/admin-api-examples#call-admin-api-from-arcscript). They are no longer callable directly. * As mentioned in the [previous release](./2024-01), some scripting operations, specifically connector operations like [x12Scan](../../connectors/x12#x12-operations), must be called through the `connector.rsc` endpoint with an auth token. The ability to call these operations directly is now disabled by default. * Certificate expiration alerts are now sent through an aggregated email that includes the details for all expiring certificates. * The dbOps scripting operations now require a license for CData drivers. * The Transactions tab in the [Activity](../administration/activity) page is now hidden by default. * The SLAs feature is now restricted to Enterprise licenses. * The [certificates](../administration/settings/settings-page#certificates) tab in the Settings page is now available to all license types. * The embedded Jetty web server that ships with the [Cross-Platform Edition](../installation/cross-platform-edition) has been updated to v10.0.20. ### Fixed * Fixed a security vulnerability affecting Jetty web servers that could lead to unauthenticated use of some application endpoints. More details can be found [here](https://www.cdata.com/kb/entries/jetty-cve-0324.rst). * Fixed an issue causing Flow APIs to not send email alerts or use the error path in the flow in some scenarios. ## Connectors ### Added * Added the ability to check the last run time of a [REST connector](../../connectors/rest) through `_connector.LastRunTimestamp` in [ArcScript](../../scripting) to allow for workflows that need to request items since the last time the connector ran from an API. Similarly, `_connector.CurrentTimestamp` evaluates to the current timestamp, which will become `_connector.LastRunTimestamp` in the next run. * Added a [Confirm Shipment action](../../connectors/amazon-marketplace/amazon-marketplace-confirm-shipment) to the Amazon Marketplace connector. * Added a new "Allow ArcScript in URL" setting to the [SOAP connector](../../connectors/soap) to allow for [ArcScript](../../scripting) evaluation in the URL setting. * Added support for the Success Path output for [FTP](../../connectors/ftp-server) and [SFTP Server](../../connectors/sftp-server) connectors. ### Changed * "Save to Sent Folder" is now False by default across all connectors. * The [Script connector](../../connectors/script) now attempts to discover external scripts in a case-insensitive manner. If two scripts exist that only differ in their casing, the exact match is used. * All CData drivers bundled with the application have been updated to the latest version. * The [Script](../../connectors/script) and [REST](../../connectors/rest) connectors now have an Action setting that defines its position in the flow. Configurations updated from previous versions will use a special "Legacy" value for this setting to ensure no behavior changes from previous versions in case a single connector was used to facilitate more than one of these actions. * The [Notify connector](../../connectors/notify) now acts as a terminal connector for new connector instances. Existing instances still behave as a transform connector as they did before. In order to achieve the same result in new connector instances, the [Success Path](../../flows/designing-a-flow#additional-configuration-options) feature should be used instead. ### Fixed * Fixed some intermittent errors that would occur in the [Webhook connector](../../connectors/webhook) under very heavy load. * Fixed an issue in the [File connector](../../connectors/file) where a greedy external process could create a lock for a file after the connector wrote the file, but before it was finalized. The connector now maintains the lock until it is finished with the file. * Fixed an issue causing the Ignore Case toggle to be ignored when using the Equal To operator in the [Branch connector](../../connectors/branch). * Fixed an issue in the [XML Map connector](../../connectors/xml-map/xml-map) where an error would be thrown in the designer if a complex expression was used for a tracked field. ### Removed * All deprecated connectors (Multi-Action Database connectors, SharePoint V1, File Match, Header Match, XML Match, and Lookup) have been removed from the application. If you are still using these connectors in an older version, please reach out to [arcsupport@cdata.com](mailto:arcsupport@cdata.com) before upgrading. # July 2024 (24.2.9039) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2024-07 Release notes for CData Arc July 2024 (build 24.2.9039). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/GvmtaSurMlw](https://www.youtube.com/embed/GvmtaSurMlw) ## Key Features ### Added * Added the ability to save [notes](../../flows/designing-a-flow#using-notes) directly in the flow designer. * Added the ability to resize connector shapes in the flow designer. * Added a [threadSleep](../../scripting/operations/op-thread-sleep) ArcScript operation. ### Changed * The certificates API now uses a date format that is consistent with other APIs like transactions, logs, requests, etc. * [Function](../../scripting/value-formatters/function-formatters) calls in ArcScript are now case-insensitive. * The Application Logs now include information about the AppDB connection during the startup sequence. * Relaxed the filename length limitation when uploading files in the UI where the application is running in environments that do not place restrictions on filename lengths. * Updated the SQLite and Derby libraries that ship with the application to their latest versions. * Security and Performance improvements across the board. ### Fixed * Fixed an issue causing errors in the dashboard for users with very high transaction volumes. * Fixed an issue with the pub/ping.rst endpoint causing errors to be reported in load balanced environments when concurrency is very high. * Improved the new flow designer's handling of special characters in workspace and connector names. * Fixed an issue in the import Admin API endpoint that caused flow connections to be lost for some imports. ## Connectors ### Added * Added a new version of the [SAP IDoc](../../connectors/sap-idoc/sap-idoc) connector. * Added the ability to upload custom schemas directly in the UI for EDI connectors. * Added the ability to reset data in a connector's state storage back to its initial state in the UI for EDI connectors. * The [Response event](../../connectors/webhook#custom-responses) in the Webhook connector now allows items to be pushed out in order to cause custom messages to be sent down the flow. * The [Email Send](../../connectors/email-send) and [Email Receive](../../connectors/email-receive) connectors now support NTLM as an authentication mechanism. * The [JSON](../../connectors/json) connector now supports the Local File Scheme setting. * Added support for the `LastRunTimestamp` and `CurrentTimestamp` attributes of the `_connector` item to the [Script](../../connectors/script) connector. * Added a **Compress HTTP Request** setting to the Advanced tab of the [REST](../../connectors/rest) connector. ### Changed * The [XML Map](../../connectors/xml-map/xml-map) connector now makes the XML Streaming capabilities more obvious through a new toggle in the Settings page. * The [FTP Server](../../connectors/ftp-server) connector now adds a FTPServer-Virtual-Path header to received messages. * Performance and stability improvements for the [Flat File](../../connectors/flat-file) connector, including the ability to indicate that the given length for fields in a position-delimited file are based on byte-length instead of character-length. * Updated the underlying dependencies for the [Apache Kafka](../../connectors/apachekafka) connector. * New instances of the [Google Drive](../../connectors/google-drive) connector now use a PATCH request to update files, instead of a combination of DELETE and POST requests. * The InitializeVendorBill stored procedure in the [NetSuite](../../connectors/netsuite/netsuite) connector now treats `ItemList` as an XML aggregate, eliminating the need for decoding the XML aggregate in subsequent steps in the flow. ### Fixed * Fixed an issue in the [Email Send](../../connectors/email-send) connector that caused the filename provided in the Content-type header to be ignored. * Fixed an issue in the [REST](../../connectors/rest) connector where certain characters in the configuration would cause the UI to behave unexpectedly. * Fixed an issue causing the Output Behavior setting in the [REST](../../connectors/rest) connector to affect the behavior of the \_response item in the Response event. * Fixed some UI issues in the [XML Map](../../connectors/xml-map/xml-map) connector. * Fixed an issue in the [Amazon Marketplace](../../connectors/amazon-marketplace/amazon-marketplace) connector that caused some connections to throw an error related to decrypting the OAuth client settings file. * Fixed an issue in the [appSendEmail](../../scripting/operations/op-app-send-email) operation's handling of attachments. * Fixed an issue preventing events from showing up in the UI for connectors that transform messages based on a schedule. * Fixed an issue in the [API](../../connectors/api/api) connector causing authtokens to be accepted in the URL in some cases, even when the setting to **Allow Auth Tokens in Url** was not enabled. * Fixed an issue in the [Zip](../../connectors/zip) connector causing the dates on extracted files to be incorrect. * Fixed an issue that prevented the afterSend event from showing in the UI for some database connector configurations. * Fixed an issue that prevented the [Email Send](../../connectors/email-send) and [Email Receive](../../connectors/email-receive) connectors from establishing new OAuth connections. * Fixed an issue that caused the **user name** field to no longer show up for new Oracle connections. * Improved the performance of the [Email Send](../../connectors/email-send) connector in attachment mode. * Fixed a backward compatibility issue in the [OFTP](../../connectors/oftp) connector's routing functionality when upgrading from legacy versions that did not contain support for multiple workspaces. * Improved the [REST](../../connectors/rest) connector's logic for refreshing tokens to avoid concurrency issues when many workers are assigned to it. * Improved the [File](../../connectors/file) connector's logic for the **Overwrite Behavior** setting when two files with the same name are being processed at the same time. * Improved the logic for the **Rotate Interval** and **Delete Interval** settings in profiles to also ensure old log files from orphaned processes are cleaned up. * Fixed an issue in the [SharePoint](../../connectors/sharepoint/sharepoint) connector that prevented it from downloading files with some special characters. * Improved the [Batch Create](../../connectors/batch-create) connector to properly handle multi-gigabyte messages. # December 2024 (24.3.9159) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2024-12 Release notes for CData Arc December 2024 (build 24.3.9159). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/9yHvufAQlfs](https://www.youtube.com/embed/9yHvufAQlfs) ## Key Features ### Added * Added the [EDI Trading Partner Management Console](../../tpmc/tpmc) to simplify the process of managing partner and document workflows. * Added [messageRead](../../scripting/operations/op-message-read) and [messageReadLine](../../scripting/operations/op-message-read-line) operations. * Added the [jsonsubtree](../../scripting/value-formatters/function-formatters#jsonsubtree-jsonpath) function. * Added disk usage information to the UI about the disk holding the Application Directory. ### Changed * Updated the login page, navigation bar, and many other elements of the UI to a new design. * Added an optional parameter to the [vault](../../scripting/value-formatters/function-formatters#vault-itemname-ifnotexists) formatter to allow it to return a "default" value if the specified vault item does not exist. * Improved the [SSO](../administration/settings/single-sign-on) login experience to be more tolerant of minor clock skews. * Updated all CData drivers included with the application to 2024. ### Fixed * Fixed an issue that could cause the embedded web server in the .NET edition to stop serving requests if the underlying Application Directory is temporarily unavailable. * Fixed an issue that prevented workspaces and connectors from loading properly in the new flow designer if they contained some special characters in the name. * Updated the PostgreSQL driver to address a potential deadlock issue when a PostgreSQL connection is used for the Application Database and the database server is unreachable for an extended period of time. * Fixed an issue that could lead to stale lock files in situations where the application is restarted and the same process id is used for the new process. * Fixed an issue that could lead to corrupt workspaces when connector folders within that workspace are missing their configuration files. * Fixed an issue in the import experience that could lead to partially successful imports when importing into a busy workspace. * Updated the MySQL driver to fix an issue with handling multibyte character sets. * Fixed an issue preventing some state tables from being indexed properly. * Fixed some issues preventing the application from running in Jetty 12. ## Connectors ### Added * Added a way to preserve headers on batch messages as they pass through the [Batch Merge](../../connectors/batch-merge) connector. * Added the ability to obtain logs from the Test Lookup Stored Procedure experience in the UI. * Added support for Team Folders (Dropbox Business) in the [Dropbox](../../connectors/dropbox) connector. * Added the ability to send emails through SendGrid in the [Email Send](../../connectors/email-send) connector. * Added support for rollover certificates to the [AS4](../../connectors/as4) connector. * Added the ability to dynamically route messages to other workspaces in the [Workspace Send](../../connectors/workspace-send) connector. * Added support for the OAuthJWT Auth Scheme in the [SharePoint](../../connectors/sharepoint/sharepoint) connector. * Added a Line Separator setting to the Settings page for the [CSV](../../connectors/csv), [TSV](../../connectors/tsv), and [PSV](../../connectors/psv) connectors. * Added a Use Virtual Hosting setting to the [Google Storage](../../connectors/google-storage) connector. * Added `EmailReceive-Attachement-Index` and `EmailReceive-Attachement-Count` headers to output messages produced by the [Email Receive](../../connectors/email-receive) connector when in Attachment mode. ### Changed * Updated the UI of the [XML Map](../../connectors/xml-map/xml-map) connector to provide a more seamless user experience, including added support for source-side editing, searching, breadcrumbs, and more. * The [Flat File](../../connectors/flat-file) connector now makes use of an XML attribute to determine the proper name for a field when it cannot be used directly as an XML element name. * Improved the logging for the [SFTP](../../connectors/sftp) and [FTP](../../connectors/ftp) connectors to explicitly state when a file on the server is ignored due to a **Receive Filter** configured on the connector. * Improved the logic in [EDI](../../connectors/edi-connectors) connectors so ACKs generated by the connector respect the **Validate that identifiers match connector configuration** setting when it's disabled. ### Fixed * Fixed an issue in the [REST](../../connectors/rest) connector's UI that prevented empty values for headers from being saved, and some special characters from being sent in requests. * Fixed an issue in the [S3](../../connectors/s3) connector that caused downloads to be skipped when **MaxReceiveFiles** is used at the same time as caching. * Fixed a backward compatibility issue that prevented existing After Send events from being processed on various connectors. * Fixed an issue that prevented the [Confirm Shipment](../../connectors/amazon-marketplace/amazon-marketplace-confirm-shipment) stored procedure from being called in the [Amazon Marketplace](../../connectors/amazon-marketplace/amazon-marketplace) connector. * Fixed an issue in the [Amazon Marketplace](../../connectors/amazon-marketplace/amazon-marketplace) connector that caused filters to not be applied correctly for some SELECT queries. * Fixed an issue that could cause [SFTP Server](../../connectors/sftp-server) to reject new connections due to a thread conflict in some rare situations. * Fixed a potential race condition that could occur in the [File](../../connectors/file) connector when the **Temp Send Extension** setting is set and **Overwrite Option** is set to *Stop*. * Fixed an issue in database connectors that could lead to a licensing error when retrieving metadata for some tables. * Fixed an issue that prevented some PSV files from being processed by the [CSV Map](../../connectors/map) connector. * Changed some [SFTP Server](../../connectors/sftp-server) error messages to be more clear when the SFTP client requests a path that the user does not have access to. * Fixed an issue in the [SharePoint](../../connectors/sharepoint/sharepoint) connector that prevented it from creating new directories in SharePoint that contain special characters. * Improved the logic in the [AS2](../../connectors/as2) connector for determining the filename extension to use based on the content type of the incoming request. * Improved the query passthrough logic in database connectors to better handle complex queries. * Fixed an issue in the [Email Send](../../connectors/email-send) connector that caused templates created in code view to open in the designer view. * Fixed an issue in the [AS4](../../connectors/as4) connector that caused some key information in the request to be encoded improperly. * Fixed an issue in the [OneDrive](../../connectors/onedrive) connector that could lead to a 401 when downloading files in some scenarios. * Fixed an issue that prevented some foreign key references from being automatically mapped in Upsert actions for database connectors. * Improved the logic for handling the Subfolder header in the [S3](../../connectors/s3) and [Zip](../../connectors/zip) connectors to better handle the case when the Subfolder header does not include a trailing path separator. * Fixed an issue in the [Email Receive](../../connectors/email-receive) connector that caused the Callback URL to not be based off of the Base URL defined in the global settings. # April 2025 (25.1.9273) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2025-04 Release notes for CData Arc April 2025 (build 25.1.9273). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/nUk3tI1vGNE](https://www.youtube.com/embed/nUk3tI1vGNE) ## Key Features ### Added * Added [2FA](../administration/settings/sign-in-settings) support. * Added [End to End testing](../../tpmc/e2e-workflow-testing) for the Trading Partner Management Console. * Updated the [Profiles](../administration/profiles) page to a new design that provides more high-level information at a glance through a new landing page, and includes lists of associated connectors and logs for each profile. * Added [cryptoEncrypt](../../scripting/operations/op-crypto-encrypt) and [cryptoDecrypt](../../scripting/operations/op-crypto-decrypt) operations to provide the ability to encrypt and decrypt data via script. * Added support for the `$filter` OData parameter to the connectors Admin API resource. * Added the ability to set custom headers in the Jetty Embedded Web Server via a new `cdata.http.customHeaders` setting in [arc.properties](../installation/cross-platform-edition#configuring-the-embedded-jetty-server). * The [workspaces grid](../../flows/workspaces#using-the-workspaces-grid) now allows for revealing the settings pane for a connector from the search results without loading the workspace. * The transactions Admin API resource now returns a ConnectorType output. ### Changed * The Jetty Embedded Web Server has been updated to Jetty 12 and now requires Java 17 or newer to run. * Improved the performance of inserting records into the transactions database. * Renamed the archiveLogs Admin API action to cleanup to better reflect its usage, and improved it to support triggering cleanup for specific workspaces and connectors. * Improved the application's health checks to trigger garbage collection before checking the available memory so as to avoid false alerts. ### Fixed * Fixed an issue in the arrayaverage formatter causing an error to be thrown when the result was a repeating decimal. ## Connectors ### Added * Added the [XML Group](../../connectors/xml-group) connector for grouping XML records by key columns. * Added the [Label](../../connectors/label) connector for generating ZPL labels. * Added the [ZUGFeRD](../../connectors/zugferd) connector for parsing and generating ZUGFeRD files. * Added a Max Records setting to the [Split](../../connectors/split) connector to control the number of records that are included in each output message. * Added support for the OAuth Auth Scheme to the [Kintone](../../connectors/kintone/kintone) connector. * Added support for the None Auth Scheme in the [Email Send](../../connectors/email-send) connector for anonymous authentication. * Added support for the ENTSOG v4 and eDelivery v2 profiles to the [AS4](../../connectors/as4) connector. * Added support for duplicate checking in [EDI connectors](../../connectors/edi-connectors). * Added a "RoundingMode" input to the [round](../../scripting/value-formatters/number-formatters#round-integer_value) formatter. * Added a "TrimResult" input to the [split](../../scripting/value-formatters/transform-string-formatters#split-delimiter-indextoreturn-trimresult-removeempty) formatter. ### Changed * Relaxed the type checking in the [JSON](../../connectors/json) connector to allow for `xli:nil` and `json:type` to be specified on the same element when translating from XML to JSON. * Moved support for ODBC drivers in the .NET Edition to be under the "Other" Driver Name option in the [Database](../../connectors/database/database) connector. * Improved the logs for the [SFTP Server](../../connectors/sftp-server) connector to aggregate logs from repeated user lockout messages, and log them as warnings instead of errors. * Added InterchangeSyntaxIdentifier and InterchangeSyntaxVersion outputs to the edifactScan operation. * Clarified the UI and documentation for the [AMQP](../../connectors/amqp) connector's "JSON" Content Type to better describe the composite data structure that it expects. * [EDI connectors](../../connectors/edi-connectors) now finalize messages from the ACK Path with a suffix on the MessageId so Output messages can reuse the same MessageId as the input message. * [Script](../../connectors/script) connectors now include the output stream from the script in the logs. * Improved the [S3](../../connectors/s3) and [Zip](../../connectors/zip) connectors to always treat the Subfolder header like a folder, regardless of whether it has a trailing path separator. ### Fixed * Fixed an issue in the [Salesforce](../../connectors/salesforce/salesforce) connector preventing OAuth access tokens from being refreshed in some situations. * Fixed an issue preventing foreign key references from automatically being mapped in some database connector configurations. * Fixed the Lookup Stored Procedure action in database connectors to wrap each result set in a new XML element instead of combining them. * Fixed an issue preventing the [messageRead](../../scripting/operations/op-message-read) operation from loading the input message within the [XML Map](../../connectors/xml-map/xml-map) connector. * Fixed an issue causing null values returned from the Lookup action in database connectors to be returned as a string. * Fixed an issue in the [IBM MQ](../../connectors/ibmmq) connector preventing some large messages from being received fully. # September 2025 (25.2.9376) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2025-09 Release notes for CData Arc September 2025 (build 25.2.9376). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/9799bYQesdw](https://www.youtube.com/embed/9799bYQesdw) ## Key Features ### Added * Enhanced concurrent user support to safely allow multiple users to access the administration console at the same time. * Added timezone display functionality showing local and server timezone information in the UI and logs. * Enhanced AWS integration with IMDS v2 support for retrieving instance metadata. * Added a `vault` API Resource to the [Admin API](../../admin-api) for managing Arc Key Vault operations programmatically. * Added Python scripting support for [ArcScript](../../scripting), allowing users to write scripts in a familiar language. * Added SAML 2.0 integration for [Single Sign-On](../administration/settings/single-sign-on) authentication. * Improved password reset functionality with enhanced forgot/reset password flow. * Enhanced RBAC capabilities with improved certificate policies and permissions management. * Added the ability for non-admin users to view the [Certificates](../administration/settings/settings-page#certificates) tab with appropriate permissions. * Added certificate search functionality by serial number in the Certificates UI. * Added timezone conversion capabilities to [ArcScript](../../scripting) for handling timezone transformations. * Added the ability to use [Workspace Send](../../connectors/workspace-send) and [Workspace Receive](../../connectors/workspace-receive) connectors in [EDI page document workflows](../../tpmc/tpmc). * Improved the visibility of [notes](../../flows/designing-a-flow#using-notes) in the flow designer. * Added a new `encoding` input for the [encEncode](../../scripting/operations/op-enc-encode) and [encDecode](../../scripting/operations/op-enc-decode) operations. * Added the ability to export EDI partner settings from the [Trading Partner Management Console](../../tpmc/tpmc) as part of the Application Settings Export wizard. * Added support for evaluating Arc Script in the global Subject field in Settings > Alerts. ### Changed * Updated .NET Framework requirement to version 4.6.2 for enhanced security and performance. * Enhanced security measures addressing various vulnerabilities in core components. * Improved file download handling, particularly addressing Safari browser compatibility issues. * Enhanced form validation and user feedback mechanisms throughout the application. * Updated autocomplete attributes on password fields for better security. * Improved the search functionality within [workspaces](../../flows/workspaces) to allow searching for connector type in addition to Connector Id. * Updated the embedded Jetty web server to version 12.0.17 for improved security and performance. * The classic flow designer is now deprecated and is scheduled to be removed in January 2026. * Updated the Java Runtime Environment (JRE) to address security vulnerabilities and improve overall system security. ### Fixed * Fixed an issue in the [Admin API](../../admin-api) where POST requests to create a workspace did not return a proper response. * Restricted application and access log viewing and downloading to cdata\_admin and cdata\_standard roles only for enhanced security. * Resolved backend database errors when handling long usernames. * Fixed various UI navigation issues in EDI management interfaces. * Fixed an issue with Two-Factor Authentication (2FA) when using a custom context path in arc.properties where login would not work as expected. * Fixed an issue where 2FA related attributes associated with exported users were not handled properly. * Fixed a Transaction tab issue where the file view modal was very small until formatting was applied. * Fixed an issue that caused incorrect transaction counts to be displayed in the Transactions tab. * Fixed an issue where clicking "Enable Automation" in Start Trigger didn't directly open the connector Automation page. * Fixed a memory leak issue that could occur in extremely high volume processing scenarios. ## Connectors ### Added * Added the ability to promote EDI metadata from files to message headers in [EDI connectors](../../connectors/edi-connectors). * Added batch upsert functionality to the [Salesforce](../../connectors/salesforce/salesforce) connector to minimize API requests and improve performance. * Added a new [DB2 connector](../../connectors/db2ibmi/db2ibmi) for iSeries/AS400 systems. * Added support for Upsert mode in the [Kintone](../../connectors/kintone/kintone) connector using the CData driver's upsert syntax. * Added a new header for overdue ACK tracking in [EDI connectors](../../connectors/edi-connectors). * Added a new [Peppol connector](../../connectors/peppol) for European B2B document exchange. * Added a new [Encoding connector](../../connectors/encoding) for character encoding transformations. * Added support for SAS token authentication in the [Azure Blob](../../connectors/azure-blob) connector. * Added support for non-ASCII characters in [encDecode](../../scripting/operations/op-enc-decode) operations for [ArcScript](../../scripting). * Added support for NTLM authentication in the [SharePoint](../../connectors/sharepoint/sharepoint) connector. * Added the ability to seamlessly reference EDI metadata in the Branch Settings of the [Branch](../../connectors/branch) connector. * Added support for filters and conditions in Excel templates, allowing more sophisticated data processing in the [Excel](../../connectors/excel) connector. ### Changed * Completely redesigned the [REST connector](../../connectors/rest) user interface with new authentication panel, request configuration, testing capabilities, and the ability to import configuration details from Swagger definitions. * Redesigned the [Form connector](../../connectors/form) with improved user interface and enhanced functionality. * Redesigned the [Script connector](../../connectors/script) with better testing support and improved user experience. * Added a new version of the [AS4](../../connectors/as4) connector with multiple improvements including better message handling and compliance features. The V1 version of the AS4 connector is now deprecated. * Improved error messaging in the [EDIFACT](../../connectors/edifact) connector, particularly for missing UNA characters. * Updated the [Email Send](../../connectors/email-send) connector to handle attachment filtering more gracefully. * Improved the [SFTP Server](../../connectors/sftp-server) connector for better file handling and connection stability. * Enhanced [XML Map](../../connectors/xml-map/xml-map) connector to use XML streaming by default until a mapping is performed that is incompatible with streaming. * The [ZUGFeRD](../../connectors/zugferd) connector has moved out of Beta status into Full release with support for processing multiple PDF attachments. * The [XML Group](../../connectors/xml-group) connector has moved out of Beta status into Full release. * The [Label](../../connectors/label) connector has moved out of Beta status into Full release. * Updated the [Webhook](../../connectors/webhook) connector to include WorkspaceId input in the Response event and modified ConnectorId values to exclude workspace information. * Updated the [Shopify](../../connectors/shopify/shopify) connector to use the GraphQL-based API by default, replacing the deprecated REST API. Existing connector instances should be transitioned to GraphQL to avoid service disruption when Shopify officially sunsets the REST API. * Improved [Google Drive](../../connectors/google-drive) connector to properly handle macros in Remote Folder Path settings. * Improved [REST](../../connectors/rest) connector XML formatting capabilities for better compatibility with other tools. * Enhanced [ZUGFeRD](../../connectors/zugferd) connector to better recognize PDF invoices and attachments from PDF metadata. * Updated various connector drivers to address security and stability issues. * Improved the handling of very large XML documents in database connectors. * Improved the handling of Excel operations when sheet names contain the "#" character, allowing proper access to sheets with special characters. * Improved the [Peppol](../../connectors/peppol) connector to properly handle Peppol IDs containing uppercase letters in SMP resolution. ### Fixed * Fixed the [XML Map](../../connectors/xml-map/xml-map) connector to prevent saving empty Conditions or Script nodes. * Resolved inconsistent behavior of the `def` formatter between various nodes in [XML Map](../../connectors/xml-map/xml-map). * Fixed [X12Scan](../../connectors/x12#x12-operations) output attributes in [ArcScript](../../scripting) to match documentation. * Resolved database lookup display issues that were not showing retry automation settings properly. * Fixed an issue where the [Email Send](../../connectors/email-send) would send an empty email when the provided attachments were excluded via filters. * Fixed various display issues in the [Flat File](../../connectors/flat-file) connector UI caused by non-zero control field indexes. * Fixed a backward compatibility issue preventing some OpenPGP keys from showing up in dropdowns in the UI. * Fixed an issue base64 decoding some attachments in the [Email Receive](../../connectors/email-receive) connector. * Fixed an issue downloading long lists of files in shared folders in the [Box](../../connectors/box) connector. * Updated CData Shopify driver to address GraphQL migration issues. * Fixed an issue in the [DB2 for IBM i](../../connectors/db2ibmi/db2ibmi) connector where the Select action failed to retrieve a list of tables in some environments. * Fixed an issue in the [AS2](../../connectors/as2) connector where copying to another workspace would cause stream related errors. * Fixed an issue in the [REST](../../connectors/rest) connector where Swagger import had formatting and parameter encoding issues. * Fixed display issues in the [XML Map](../../connectors/xml-map/xml-map) connector where renaming XML nodes was aborted too easily. * Fixed an issue where connector settings pages would take too long to open for connectors that use the new UI (Form, Script, REST). * Fixed the new [Script](../../connectors/script) connector UI to also show automation retry settings in the Automation tab when action is set to Transform. * Fixed an issue in the [XML Map](../../connectors/xml-map/xml-map) connector where emails were not correctly sent to overridden email addresses in Add Mapping Conditions. * Fixed [Azure Blob](../../connectors/azure-blob) connector download failures for storage accounts with hierarchical namespace enabled. * Fixed an issue preventing EDI workflow routing for Flow API calls where 997 EDI acknowledgments could not close pending MDNs. * Fixed [XML Map](../../connectors/xml-map/xml-map) connector backward compatibility issues with old script node values. * Fixed a backward compatibility issue in the [REST](../../connectors/rest) connector that affected output filenames for transform actions. * Updated the underlying SFTP component library to resolve a casting error in the [SFTP connector](../../connectors/sftp) that was causing connection failures when certain algorithms were selected. * Fixed an issue in the [Google Storage](../../connectors/google-storage) connector that prevented the connector from interacting with certain buckets. * Resolved a connection issue in the [Google Storage](../../connectors/google-storage) connector where the "Connect" button would fail unless connector settings were explicitly saved first. * Fixed the [Zip](../../connectors/zip) connector to properly preserve file timestamps when operating in Scheduled Send mode. # December 2025 (25.3.9469) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2025-12 Release notes for CData Arc December 2025 (build 25.3.9469). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/gv8QuAHgCIA](https://www.youtube.com/embed/gv8QuAHgCIA) ## Key Features ### Added * Added a comprehensive redesign of the [Settings](../administration/settings/settings-page) page with improved user interface and enhanced functionality across all tabs including Users, Roles, Certificates, Connections, Security, Vault, License, and Advanced settings. * Added comprehensive security improvements including password history, expiration policies, forced admin resets, IP whitelist/blacklist functionality, and automatic account disabling for inactive users. * Added connector creation date tracking to provide better visibility into when connectors were originally created. ### Changed * Improved support for [FIPS 140-2 compliance](../administration/security-best-practices#fips-compliance) to meet stringent federal security requirements for cryptographic operations. - Improved [User Roles](../administration/settings/user-roles) to relax the maximum role restriction for a user, and the maximum policy restriction for roles. - Improved message display functionality in the Arc Message viewer with better organization of repeated headers and enhanced readability. - Updated the arc.properties file to support system property and environment variable expansion for more flexible configuration management. - Adding paging to the Logs API resource in the Admin API. - Added new inputs to the export and import API actions to provide more granular control. - Improved the performance of the cleanup procedure. - Improved the performance of the application database by leveraging partitioning when SQL Server is used. - Public certificates are no longer generated while importing private certificates. Instead, they can be exported from private certificates directly in the UI. ### Fixed * Fixed the files resource in the Admin API to properly handle filenames containing commas. * Fixed ArcScript \_cookie item enumeration to properly iterate through all available cookies. * Fixed transaction tab filtering to properly handle combined direction and status filters. * Fixed certificate handling to properly manage certificates with identical names but different expiration dates. * Fixed a UI issue preventing the setup of Ollama as an AI provider when the selected instance only included one model. * Fixed a memory leak in the .NET edition's locking implementation. * Fixed a database connection leak in the application's health check routine. * Many performance and stability improvements to the cleanup routine, especially when archiving to S3. * Fixed an issue migrating transactions to the new version of the underlying table for derby databases with a large number of transactions. * Fixed a backward compatibility issue in the certificates API resource where the format of the ConnectorIds output changed. This has been reverted to restore backward compatibility, but is now considered deprecated and will be removed in a future version in favor of the new AssociatedResources output. * Updated the JRE that ships with the Java edition to address security vulnerabilities in the old JRE. ## Connectors ### Added * Added AI capabilities to the [XML Map](../../connectors/xml-map/xml-map) connector to automatically generate mappings. * Added [ICAP connector](../../connectors/icap) for seamless integration with ICAP servers to perform real-time content adaptation and security scanning. * Added HMAC signature authentication support for the [Webhook](../../connectors/webhook) connector, providing stronger authentication options for Professional and Enterprise license tiers. * Added a migration utility to help users transition from deprecated AS4v1 connector instances to the improved AS4v2 implementation. Users looking to upgrade [AS4](../../connectors/as4) connector configurations in bulk should reach out to [arcsupport@cdata.com](mailto:arcsupport@cdata.com) for assistance. * Added Order By capabilities to the [XML Group](../../connectors/xml-group) connector. * Added a display density toggle in the [XML Map](../../connectors/xml-map/xml-map) connector. ### Changed * Improved the [AS4](../../connectors/as4) connector's business profile handling with better settings synchronization and configuration management. * Enhanced the [Google Drive](../../connectors/google-drive) connector with proper macro resolution in remote folder paths for dynamic folder creation. * Updated the [Form](../../connectors/form) connector with improved default value handling and better validation for optional fields. * Enhanced the [Webhook](../../connectors/webhook) connector with improved request limitation error reporting and better resource usage warnings. * Improved the [SFTP Server](../../connectors/sftp-server) connector with better authorization failure handling and reduced log verbosity for repeated failures. * Updated the [Script](../../connectors/script) connector with enhanced test code functionality and improved the display of repeated headers. * Improved the performance of all database style connectors for handling large XML encoded CLOB data with significantly improved parsing speed. * Improved [AS2 connector](../../connectors/as2) error messaging related to ECC keys. * Improved the performance of the [Script](../../connectors/script) connector. * The [Peppol](../../connectors/peppol) connector has moved out of Beta status into Full release. * The [Encoding](../../connectors/encoding) connector has moved out of Beta status into Full release. * The [QuickBooks Online](../../connectors/quickbooks-online/quickbooks-online) connector's underlying OAuth credentials were updated. As a result, connections should be reauthenticated. * The [Apache Kafka](../../connectors/apachekafka) connector's underlying dependencies were updated to address security vulnerabilities. * Improved the performance for switching tables in cdata-based database connectors. ### Fixed * Fixed [SharePoint connector](../../connectors/sharepoint/sharepoint) document library access issues related to many edge cases like renamed document libraries, folders containing whitespace or special characters in the name, and improved error messaging. * Fixed the [REST](../../connectors/rest) connector UI's handling of duplicate header names in the configuration. * Fixed [XML Map](../../connectors/xml-map/xml-map) connector template file display and schema linking issues when copying between workspaces. * Fixed various UI display issues across multiple connectors including table formatting, modal behavior, and responsive design improvements. * Improved Swagger import functionality in [REST](../../connectors/rest) connector to properly include base URLs and response content types. * Fixed [Flat File](../../connectors/flat-file) connector configuration persistence for files with Control Field Index greater than 0. * Fixed an issue in the [X12](../../connectors/x12) connector that could lead to the inability to load embedded schemas when the application is run in endpoint-only mode. * Fixed an issue in the [Azure Blob](../../connectors/azure-blob) connector causing duplicate file downloads with storage accounts using hierarchical namespaces when "Recurse remote subdirectories" is enabled and caching is disabled. * Fixed an issue preventing new OAuth connections from being established in the [Square](../../connectors/square/square) and [Dynamics 365 Business Central](../../connectors/dynamics-365-bc/dynamics-365-bc) connectors. * Fixed an issue preventing new connections using Consumer Key and Consumer Secret from being established in the [Woo Commerce](../../connectors/woocommerce/woocommerce) connector. * Files uploaded with the [Google Drive](../../connectors/google-drive) connector now preserve the last modified time of the file being uploaded. * Fixed an issue in the [SharePoint](../../connectors/sharepoint/sharepoint) connector related to usage of the "Use Entity Type Name" configuration setting, which has now been removed. * Fixed an issue in the [SOAP](../../connectors/soap) connector causing an extra semicolon to be added to the Content-Type header. * Fixed an issue in the [AS4](../../connectors/as4) connector that caused errors to be thrown when attempting to pull files from an empty queue. * Fixed an issue in the [Google Storage](../../connectors/google-storage) connector preventing files from being downloaded when Receive Remote Folder was set in some situations. * Fixed an issue preventing the [Branch](../../connectors/branch) connector from properly parsing custom datetime formats. # April 2026 (26.1.9515) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2026-04 Release notes for CData Arc April 2026 (build 26.1.9515). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/95DjPDF0mL4](https://www.youtube.com/embed/95DjPDF0mL4) ## Key Features ### Added * Added [Git Version Control](../administration/git-version-control) support to enable versioning of Arc configurations through integration with Git repositories. This Enterprise-only feature allows teams to track changes, collaborate on configurations, and synchronize settings across environments. * Added [Tamper Evident Logging](../administration/activity#tamper-evident-logging) to provide cryptographic verification that audit logs have not been modified, ensuring audit trail integrity for compliance requirements. * Added dynamic EDI partner relationship tracking for EDI connectors to better manage and visualize trading partner connections. * Added the ability to [download and update CData drivers](../administration/settings/connections) directly from within the application, simplifying the process of keeping drivers current. * Added *exportDataEncryptionKey* and *importDataEncryptionKey* actions to the [Admin API](../../admin-api) to facilitate secure key transfer between environments when setting up Git Version Control. * Added a Connectors view in the [Flows](../../flows) page to provide an alternative way to browse and manage connectors across workspaces. The legacy connectors.rst endpoint is now deprecated and will be removed in a future release. * Added binary I/O support for the [Flow API](../../flows/flow-api), enabling direct handling of binary data in API requests and responses. * Added support for the [Workspace Send](../../connectors/workspace-send) connector in Flow API configurations. * Added Tracked Headers support in [Reports](../administration/reports) with a redesigned user interface for easier configuration and visualization. * Added the ability to preview common file types (EDI, CSV, IDoc, etc.) in a "print view" in the UI, as opposed to the raw file contents. * Added a global option to disable [SLA](../administration/activity#slas-tab) alerts. * Added support for additional LLM providers including Gemini, Deepseek, and Ollama Cloud for AI-powered features. * Added a *requeueMessage* action to the [Admin API](../../admin-api) for reprocessing messages. ### Changed * Improved the way Arc encrypts sensitive values. This is enabled by default for all new installations, as well as existing Java installations that do not already have an arc.properties file. Users of Arc that do not meet these criteria should review [Defining a Custom Master Key](../installation/custom-master-key) to learn how to opt in. * Updated the embedded Jetty web server to version 12.0.29 for improved security and performance. * Simplified the procedure for running endpoint-only versions of Arc. As a result, the Cross-Platform Edition no longer ships with services.war. It can be generated from arc.jar, as documented [here](../administration/split-plane-deployment#setting-up-the-endpoint-only-application-2). * The Cross-Platform Edition's arc.war now requires Java 17 or above to run. As a result, traditional WebSphere is no longer supported. Use [WebSphere Liberty](../installation/cross-platform-edition#configuration-in-websphere-liberty) instead. * Improved the application health check routine to check available inodes on Linux systems. * Changed the default value of the **HTTP Forwarded Header** setting to be empty to prevent spoofing attacks in environments where it is not required. * Removed the limitations around the number of roles that can be associated with a user, and the number of policies that can be associated with a role. * Added the ability to specify CC email addresses for the [appSendEmail](../../scripting/operations/op-app-send-email) operation. * Improved report emails to more closely mirror the report that is displayed in the UI. * Improved the performance for all incoming HTTP requests, especially for the Flow API. * All CData drivers have been updated to v25. ### Fixed * Improved the handling of IPv6 addresses when auto-generating Base Url. * Improved the cleanup routine to better handle very large volumes of files, especially when archiving to S3. * Updated the JRE that ships with the Cross-Platform Edition to resolve security vulnerabilities. * Updated the Redis driver to improve performance and fix memory leaks that could occur in some situations. * Resolved an issue preventing [sysExecute](../../scripting/operations/op-sys-execute) from capturing script output during high-concurrency scenarios. ## Connectors ### Added * Added an [AI Connector](../../connectors/ai) for integrating with AI models. * Added the ability to set and preserve cookies in the [REST](../../connectors/rest) connector. * Added the ability to override AI prompts in the mapping editor for customizing AI-assisted mapping behavior. * Added a new UI for uploading test files in connectors that support it: [CSV](../../connectors/csv), [TSV](../../connectors/tsv), [PSV](../../connectors/psv). * Added support for Local File Scheme to the [ZUGFeRD](../../connectors/zugferd) connector. * Added MessageKeys support to the [Apache Kafka](../../connectors/apachekafka) connector for better message routing and partitioning control. * Added a **Follow Authorization Header** setting to the [REST](../../connectors/rest) connector for controlling how authorization headers are handled during redirects. * Added a **TLS Enabled Protocols** setting for the [Email Send](../../connectors/email-send), [Email Receive](../../connectors/email-receive) and [MQTT](../../connectors/mqtt) connectors. * Added a **Propagate Merged Cell Values** setting to the Advanced tab of the [Excel](../../connectors/excel) connector. ### Changed * The [ICAP](../../connectors/icap) connector has moved out of Beta into Full release status. * The [AS3](../../connectors/as3), [Batch Create](../../connectors/batch-create), [Batch Split](../../connectors/batch-split), [Box](../../connectors/box), [Copy](../../connectors/copy), [CSV](../../connectors/csv), [Dropbox](../../connectors/dropbox), [Google Drive](../../connectors/google-drive), [Google Storage](../../connectors/google-storage), [JSON](../../connectors/json), [OneDrive](../../connectors/onedrive), [PIP](../../connectors/pip), [PSV](../../connectors/psv), [RNIF](../../connectors/rnif), [S3](../../connectors/s3), [Schedule](../../connectors/schedule), [Split](../../connectors/split), [TSV](../../connectors/tsv), and [WebDAV](../../connectors/webdav) connectors were all updated to a new UI design. * [FTP](../../connectors/ftp) connectors now require the size of downloaded files to be no greater than 3x the size reported by the file listing by default. * Improved the logging for the [SFTP Server](../../connectors/sftp-server) connector to recognize repeated log events and aggregates them into a single message. * Improved logging for OAuth based connections. * Improved the UI for the [Script](../../connectors/script) connector to better reflect the inputs that are available according to the selected Action. * Improved the UI for the [Script](../../connectors/script) connector to inject an [arc:push](../../scripting/keyword-reference/op-arc-push) keyword when an output attribute is clicked if none are already present. * Improved the UI for the [REST](../../connectors/rest) connector to remember previously used Swagger URLs. * Previously deprecated Shopify schemas have now been removed from the [Shopify](../../connectors/shopify/shopify) connector. * The [Amazon Marketplace](../../connectors/amazon-marketplace/amazon-marketplace) connector's Update Inventory, Submit Vendor Order Acknowledgement, Submit Vendor Shipment Confirmations, and Submit Vendor Payments actions now require the Vendor Central schema. ### Fixed * The [PSV](../../connectors/psv) connector appropriately respects the Force Cell Delimiter setting. * Fixed an issue in [CSV Map](../../connectors/map) preventing Code View from displaying properly for some legacy configurations migrated from older versions. * Fixed an issue in the [REST](../../connectors/rest) connector preventing the connector from promoting response headers if **Output Behavior** is set to Input Message. * Fixed the [XML Map](../../connectors/xml-map/xml-map) connector to use the output message when evaluating macros in **Local File Scheme**. * Fixed the [JSON](../../connectors/json) connector to validate that number types are actually numbers. * Fixed an issue preventing the `_message` item from being available in custom events provided by the [REST](../../connectors/rest), [Webhook](../../connectors/webhook), and [TCP Server](../../connectors/tcp-server) connectors. * Fixed an issue preventing the `_message` item from being available in the [SOAP](../../connectors/soap) connector's **Service Endpoint** setting. # July 2026 (26.2.9686) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2026-07 Release notes for CData Arc July 2026 (build 26.2.9686). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/pq9\_p3hllX8?si=GqiXyVuih1L4Ubtn](https://www.youtube.com/embed/pq9_p3hllX8?si=GqiXyVuih1L4Ubtn) ## Key Features ### Added * Added OAuth 2.0 (Client Credentials flow) as an option for [authenticating to Arc's public API endpoints](../../admin-api/admin-api-authentication) (Admin API, Flow API, Webhook, and so on). * Added the ability to control HTTP status codes and response headers in the [Flow API](../../flows/flow-api) via the `API-Response-StatusCode` and `API-Response-Header-{HeaderName}` message headers. * Added support for triggering a receive as the first step in a [Flow API](../../flows/flow-api) flow. * Added the ability to promote incoming HTTP header values as metadata on messages in the [Flow API](../../flows/flow-api). * Added a Flow API resource to the [Admin API](../../admin-api/admin-api-resources). * Added the ability to establish [Version Control](../administration/git-version-control) connections via OAuth and SSH directly in the UI. * Added the ability to restore files from a commit in the version control UI. * Added the ability to delete local branches in the version control UI. * Added audit log entries for version control operations. * Added the ability to control the Max Files Per Hour that flow through the [Schedule](../../connectors/schedule) connector. * Added a [Discard](../../connectors/discard) connector to provide a clean terminal point for flows that produce unwanted messages. * Added the ability to filter by tracked headers in the Transactions tab of each connector. * The [md5hash](../../scripting/value-formatters/encoding-decoding-string-formatters#md5hash-encodetobase64-inputencoding), [sha1hash](../../scripting/value-formatters/encoding-decoding-string-formatters#sha1hash-encodetobase64-inputencoding), and [sha2hash](../../scripting/value-formatters/encoding-decoding-string-formatters#sha2hash-algorithm-encodetobase64-inputencoding) formatters now support binary content by accepting a new encoding parameter (`utf-8`, `hex`, or `base64`) for the input data, enabling use cases like hashing image files. * Added the ability to migrate [CData](../../connectors/cdata/cdata) connector instances to their new dedicated connector types. The CData connector is now deprecated. * New installations of the Arc Cross-Platform Edition now use H2 as the default database. * Added an `archive` folder within the Application Directory as the default archive location for new installations. Existing configurations without **Archive Location** set will continue using each connector's local "Archive" folder, but this behavior is now deprecated; **Archive Location** is required going forward and connector-specific archive folders will be removed in a future release. ### Changed * Improved application database performance through partitioning when PostgreSQL is configured as the backend database. * Improved the performance of the application database by automatically determining the partition interval when SQL Server or PostgreSQL is used. * Changed the default value for **Log Messages** to be `False` in connectors where messages simply pass through (Branch, Copy, Notify, and so on). * Changed the default value for **Log Level** to be `Error` in connectors that simply route messages in the flow (Copy, Validate, Workspace Send, and so on). ### Fixed * Corrected the response code returned by the server when attempting to access a public URL endpoint associated with a connector that does not exist. * Fixed an issue that occurred in some [Script](../../connectors/script) contexts when processing messages with more than 32,768 duplicate header values by gracefully discarding the earliest excess values and logging a warning instead of failing. * Fixed [SSO JIT provisioning](../administration/settings/single-sign-on#just-in-time-jit-user-provisioning) so that removing a user from their identity provider group now correctly revokes their application access. ## Connectors ### Added * Added new versions of the [REST](../../connectors/rest), [Email Send](../../connectors/email-send), and [Email Receive](../../connectors/email-receive) connectors with improved UIs, support for shared connections, and more. Existing connectors of these types can be upgraded automatically by clicking the upgrade button in the connector UI or the connector's shape in the flow designer. * Added a configurable **Response Event Error Behavior** setting to the [REST](../../connectors/rest) connector that controls how exceptions thrown in the Response event are handled. * Added Claude and Grok to the list of supported AI providers in the [AI](../../connectors/ai) connector. * Added the ability to use SendGrid in the [Email Send](../../connectors/email-send) connector. * Added a **Maximum File Size** setting in the [FTP](../../connectors/ftp) and [SFTP](../../connectors/sftp) connectors to avoid downloading files above a size threshold. * Exposed connection pooling related settings for the [SQL Server, PostgreSQL, and MySQL](../../connectors/database/database) connectors and changed the default behavior to enable connection pooling. * Added the ability to specify a collection of related XSD files to be used in the [XSD](../../connectors/xsd) connector. * Updated [MFT connectors](../../connectors/mft-connectors) with caching enabled to automatically prune stale entries from the cache. ### Changed * The [AI](../../connectors/ai) connector is now generally available and is no longer marked as a beta connector. * Changed the default behavior of the [Email Receive](../../connectors/email-receive) connector to skip processing of messages marked for deletion on the IMAP server. * Updated the [AMQP](../../connectors/amqp) connector to align with the AMQP protocol specification for content type handling. Users who previously relied on the Binary value type to carry content type on outgoing messages must update their configuration to use the Data value type instead. * **Java edition only.** Due to an issue in the underlying SFTP component, [SFTP](../../connectors/sftp) connectors that use the **Timestamp Comparison** setting to only download new or changed files may download previously processed files again after upgrading: * **Upgrading from build 8731 or earlier, or from a 25.3 or 26.1 build numbered 9553 or later:** disable **Use GMT File Time** in the Advanced tab of the SFTP connector before the first poll after the upgrade, and leave it disabled. * **Upgrading from build 8732 to 9552 (including 25.3.9469 and 26.1.9515):** leave **Use GMT File Time** enabled. Depending on the time zone of the Arc host, some or all previously downloaded files may be downloaded once. In US time zones, these are files last modified during daylight saving time. * **Upgrading from 26.2 or later:** no action is needed. * The .NET edition is not affected. ### Fixed * Fixed several issues in the [SharePoint](../../connectors/sharepoint/sharepoint) connector that prevented subfolders from being automatically created in non-English SharePoint sites. * Fixed an issue preventing aggregate columns from being properly represented as XML in the lookup action of [database](../../connectors/database/database) connectors. # October 2026 (26.3.9783) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/2026-10 Release notes for CData Arc October 2026 (build 26.3.9783). ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/9n1lSw-gUAU?si=1UAru6n\_hzXLr5jf](https://www.youtube.com/embed/9n1lSw-gUAU?si=1UAru6n_hzXLr5jf) ## Key Features ### Added * Added support for application-level [Encryption at Rest](../administration/settings/encryption-at-rest) using AES-256-GCM, with configurable modes ranging from Compatibility to Strict. * Added support for standalone local [Version Control](../administration/git-version-control) without requiring a remote Git repository. * Added support for [Application Database Migration](../administration/settings/migration), allowing users to migrate data when switching between database backends. * Added a built-in [Remote MCP Server](../../admin-api/remote-mcp-server), allowing users to connect AI clients (Claude, ChatGPT, Copilot) directly to an Arc instance for diagnostic queries. More tools will be added in future versions. * Added role-based access control for [Connections](../administration/settings/connections), allowing administrators to scope connection permissions through [policies](../administration/settings/user-roles). * Added dedicated user management and RBAC for the [Flow API](../../flows/flow-api). * Added support for service accounts that can invoke API endpoints without application console access. * Added Log Pattern Analysis to the [Activity](../administration/activity) page, grouping similar log entries together to help identify repeated issues and noisy log patterns. * Added a `moveConnector` [Admin API](../../admin-api/admin-api-actions) action to move connectors between workspaces while preserving configuration and transaction history. * Added service status indicators to individual connectors that host servers. * Expanded the list of available connection types to include all [CData API Profiles](https://www.cdata.com/apidriver/download/). ### Changed * Previous versions of message files are now preserved on disk when a requeued or retried message would otherwise overwrite an existing file. * Improved the error message displayed when a connection to a remote Git repository fails. * Improved the [Version Control](../administration/git-version-control) diff view for settings changes. * Deprecated Sent Folder settings are now hidden when unused, and clearly marked as deprecated when still configured. * Removed the previously deprecated classic settings UI. All settings are now managed through the current settings UI. ### Fixed * The [Version Control](../administration/git-version-control) toolbar now correctly disables connection-dependent actions when the remote is unreachable. * Fixed the Download Logs button not being visible on the Error Details page. * Fixed the Test Connection dialog so the Download Logs button is available when the error message is displayed in a modal. ## Connectors ### Added * Added a [Python](../../connectors/python) connector. * Added an [AES](../../connectors/aes) connector for native encryption and decryption of message content, supporting AES-128/192/256 with multiple cipher modes. * Added the ability to relocate processed email messages to an archive folder in the [Email Receive](../../connectors/email-receive) connector. * Added a Start Time setting to schedule-based automation in applicable connectors, along with read-only Last Run Time and Next Run Time displays. Changing a scheduled time to a value that has already passed no longer triggers an immediate execution. * Added a designer for building JOIN queries in [Database](../../connectors/database/database) connectors to improve the experience with rate-limited APIs. * Added the `xsubtree` function to the [XML Map](../../connectors/xml-map/xml-map) connector for copying source content as-is during mapping. * Added a `_state` item accessible anywhere [ArcScript](../../scripting/arcscript) can be written, providing scripting access to the connector's state database without requiring API authentication. * Added support for the DateFormat macro in the [SFTP](../../connectors/sftp) and [FTP](../../connectors/ftp) connector Download and Upload path settings. * Added a Finalize on Disconnect setting to the [SFTP Server](../../connectors/sftp-server) connector, ensuring files are fully received before being moved to the next connector. * Added a Home Directory setting to the [FTP Server](../../connectors/ftp-server) and [SFTP Server](../../connectors/sftp-server) connectors, allowing administrators to specify the initial directory for connected users. ### Changed * AI-assisted mapping in [XML Map](../../connectors/xml-map/xml-map) now supports complex composite mappings that go beyond simple 1:1 element relationships. * Importing a new source in the [XML Map](../../connectors/xml-map/xml-map) connector now preserves the existing mapping instead of restarting it. * Added a warning banner in the UI to alert users when a [Salesforce](../../connectors/salesforce/salesforce) connector using Basic authentication is detected, as Salesforce has recently deprecated this authentication method. * Moved the "Allow [ArcScript](../../scripting/arcscript) in …" settings from the Advanced tab to the relevant field in all applicable connectors for better discoverability. * The Upgrade Connector workflow now allows users to specify a custom Connector Id for the upgraded connector. ### Fixed * Fixed an issue in the [Database](../../connectors/database/database) connector where selecting an update column that was not among the selected columns would cause the configuration to be lost on reload. * Upgrading connectors that were previously multi-action to their newer single-action replacement now produces a connector for each action. * Fixed an issue where the [REST](../../connectors/rest) connector included quote characters in the message filename when the server returns a quoted `Content-Disposition` filename. * Added a toast notification confirming a successful connection in the [Email Receive](../../connectors/email-receive) connector. * Fixed an issue where tables with long names could not be deleted from the [Database](../../connectors/database/database) connector UI. * Fixed [SharePoint](../../connectors/sharepoint/sharepoint) receive polling on large document libraries hanging or failing due to the List View Threshold. * Fixed an issue where the `_ctx.log` function of the [Python](../../connectors/python) engine was not writing entries to the application log. * Fixed the [Email Send](../../connectors/email-send) v2 connector not remembering Code View as the preferred editor mode and not rendering code formatting correctly. * Fixed an issue in the [Admin API](../../admin-api/admin-api-resources) `flowAPIs` resource where `WorkspaceId` alone was used as the OData entity key, which could return an invalid response when a workspace contained multiple Flow APIs. The key for `flowAPIs` is now a composite of `WorkspaceId`, `Name`, and `Method`. Breaking change: Single-entity lookup using only `WorkspaceId` (for example, `flowAPIs(Default)`) is no longer supported; to retrieve all Flow APIs in a workspace, use a filter-style request such as `flowAPIs/?WorkspaceId=Default`. * Fixed the [SCP](../../connectors/scp) connector's send action to append the Subfolder header to the destination path when uploading files. Additionally, fixed the receive action to set the Subfolder header relative to the Remote Path, excluding the root path name. # Current Release Source: https://docs.arc.cdata.com/26.4/self-hosted/en/getting-started/release-notes/current Release notes for the current version of CData Arc. ## Overview Watch this video to see a summary of what's new in this release: [https://www.youtube.com/embed/hWZ0Z29TzXE](https://www.youtube.com/embed/hWZ0Z29TzXE) ## Key Features ### Added * Expanded the [Remote MCP Server](../../admin-api/remote-mcp-server) with tools to trigger a connector, requeue a message, clean up files, and find service logs, and expanded the `diagnose_message` tool. * Added a Diagnose this error button to the transaction detail view that triggers an AI-powered analysis of a failed transaction using the configured [AI provider](../administration/settings/ai-settings). * Added an Analyze Worker Pool tool to [Performance Settings](../administration/settings/performance-settings) that captures worker pool logs over a chosen time window and uses the configured AI provider to recommend adjustments. * Added support for external secret providers in the [Vault](../administration/settings/global-settings-vault), allowing vault items to be sourced from environment variables and [Azure Key Vault](../administration/settings/global-settings-vault#adding-an-azure-key-vault-provider) (Enterprise tier) in addition to the local vault. * Added support for vault references in the [Cleanup Settings](../administration/settings/cleanup-settings) and [Additional Settings](../administration/settings/additional-settings) sections of the global Settings page. * Added support for per-workspace Git repositories for [Version Control](../administration/git-version-control), including a repository list in the version control footer and repository filtering in Commit History. * Added more detailed audit log entries that record which setting changed, along with its previous and new values, across more of Arc's configuration (connectors, workspaces, users, roles, and more). * Added a `moveConnectors` [Admin API](../../admin-api/admin-api-actions) action that moves a group of connectors to another workspace while preserving the connections between them. * Added a `users` [resource](../../admin-api/admin-api-resources) and a `refreshAPICredential` [action](../../admin-api/admin-api-actions) to the Admin API for managing users and regenerating auth tokens and client secrets. * Added a [Message Directory](../installation/arc-properties-configuration#cdata-app-messagedirectory) setting (`cdata.app.messageDirectory` in Java, `MessageDirectory` in .NET) to store message data separately from the Application Directory. * Added [Automation Logs](../administration/activity#automation-logs) to the [Activity](../administration/activity) page, recording scheduled poll attempts including polls that return no data and skipped schedules. Also added Automation Status and Last Processed Result displays to the Automation tab to show whether a connector is running, idle, or suspended, and the outcome of its last run. * Added connection status indicators (Connected, Warning, Failed, and Not Tested) to the shared connection selector in connector settings. * Added a [Dashboard](../administration/dashboard) card showing the number of connectors currently in backoff, with a link to view them in the Connectors list. * Added a configurable auto-refresh interval to the main [Dashboard](../administration/dashboard), set to every 60 seconds by default. * Added warnings in the UI when the application database approaches its defined record threshold for the configured database type. * Added application log entries when a connector's SLA transitions to At Risk or Violated, so monitoring tools can ingest SLA events without relying on email. * Added a Download Message button to the Message Log message detail view, and made the connector names in the message flow links that open the connector's settings. * Added an optional input encoding to the [base64encode](../../scripting/value-formatters/encoding-decoding-string-formatters#base64encode-inputencoding) and [hexencode](../../scripting/value-formatters/encoding-decoding-string-formatters#hexencode-uppercase-inputencoding) formatters and an optional output encoding to the [base64decode](../../scripting/value-formatters/encoding-decoding-string-formatters#base64decode-outputencoding) and [hexdecode](../../scripting/value-formatters/encoding-decoding-string-formatters#hexdecode-outputencoding) formatters, enabling operations such as generating a Base64 GUID with `[guid(false) | base64encode('hex')]`. ### Changed * Updated all connectors that use CData drivers to use the v26 drivers instead of v25. * Restricted all [Version Control](../administration/git-version-control) operations, including viewing version control logs, to administrators. Non-administrator users no longer see the Version Control options in the UI. * Made it easier to convert a [Version Control](../administration/git-version-control) repository from local to remote by switching the configuration type directly in the UI. * The [Version Control](../administration/git-version-control) diff view now includes a "View with all settings" option that shows unchanged settings alongside the changes. * Increased the maximum number of files that can be uploaded at once from a connector's Transactions page from 99 to 256. Uploads over the limit are now blocked with a message stating the maximum. * Removed the previously deprecated classic Activity page, classic flow designer, and classic Reports UI. * Added memory protection for XML processing in the [XML Map](../../connectors/xml-map/xml-map), Branch, Validate, and [Email Send](../../connectors/email-send) connectors, which now log a warning, or reject the message, when an input is too large to safely load into memory. * Clarified the next steps shown after migrating the Application Database on .NET, including the `AppDb` connection string entry to add to `Web.config`. * The `$select` parameter on the [Admin API](../../admin-api/admin-api-resources) `connectors` resource now returns connector-specific properties, such as `as2identifier`, when using the `@json` response format. ### Fixed * Fixed an issue with the default Trusted IPs setting for the [Form](../../connectors/form) and [TCP Server](../../connectors/tcp-server) connectors. * Fixed vault references not being available in workspace settings. * Fixed an issue where importing Vault items from a .arc file assigned them new Ids, which caused settings pulled in through [Version Control](../administration/git-version-control) to lose their Vault references. * Fixed an issue where some configuration was not preserved when generating the `arc.xml` file for Java installations. * Fixed an issue in the [Admin API](../../admin-api/admin-api-filtering) where an unparenthesized `not` in a `$filter` expression applied to the rest of the expression, causing incorrect results. * Fixed an issue where the [Admin API](../../admin-api/admin-api-resources) `reports` resource ignored `$filter` in `$count` requests. * Fixed the `$orderby` parameter being ignored for the [Admin API](../../admin-api/admin-api-resources) `connectors` resource. * Fixed an issue where importing a connector over an existing one did not clear settings that were empty in the imported file. * Fixed an issue on case-sensitive file systems where certificates with names that differ only by case were not all available in connector certificate selectors. * Fixed the Java installer and upgrade script not reading the `cdata.app.directory` value from `arc.properties` when it contained spaces, quotes, or escaped backslashes. * Fixed an issue where a failed push during Commit and Push in [Version Control](../administration/git-version-control) could remove local changes from the workspace. ## Connectors ### Added * Added a [Microsoft 365 Email](../../connectors/ms365email/ms365email) connector, built on the CData Microsoft 365 driver, that sends and receives email through the Microsoft Graph API with support for app-only authentication to shared mailboxes. * Added a [Partner Mailbox](../../connectors/partner-mailbox) connector that gives trading partners a self-service web portal to download files staged for them and upload files to send. * Added a [Finalize](../../connectors/finalize) connector, along with deferred finalization in the [File](../../connectors/file) connector, that delays post-processing actions such as deleting or moving source files until downstream processing succeeds. * Added an Output Behavior setting and a Response Event to the [SOAP](../../connectors/soap) connector. * Added an Encryption Algorithms setting to the [SFTP Server](../../connectors/sftp-server) connector's profile for specifying the allowed encryption algorithms. * Added the ability to compare a value against another XPath, header, or variable in the [XML Map](../../connectors/xml-map/xml-map) connector's Condition Editor. * Added support for XML lookahead syntax in the Lookup action of [Database](../../connectors/database/database) and other data-type connectors. * Added a Root Element Name setting to the [Excel](../../connectors/excel) connector to override the sheet name used as the root element of the output. * Added a Sheet Names setting to the [Excel](../../connectors/excel) connector's Advanced Settings, available in Table mode, to specify which sheets in a workbook produce output. Leaving the setting empty outputs all sheets. * Added the remote path of the uploaded blob as an input to the [Azure Blob](../../connectors/azure-blob) connector's AfterSend event. * Added a FileId input to the [Box](../../connectors/box) connector's AfterSend event. * Added a Log MDNs setting to the [AS2](../../connectors/as2) connector that keeps a copy of each MDN receipt regardless of the Log Level. The setting is enabled by default and can be turned off. * Added an Incoming Headers setting to the [Webhook](../../connectors/webhook) connector to promote specified incoming HTTP headers as message headers. * Added the ability to customize AI prompts for [Database](../../connectors/database/database) and data-type connectors that use AI-assisted mapping. * Added support for BatchGroups as input to the [File](../../connectors/file) and [S3](../../connectors/s3) connectors, along with new [ArcScript](../../scripting/arcscript) operations for working with BatchGroups. * Added a confirmation prompt in the [XSD](../../connectors/xsd) connector when an uploaded file would overwrite an existing XSD file. ### Changed * The [File](../../connectors/file) connector now receives files in ascending last-modified order. * The [AS2](../../connectors/as2) connector now signs the MDN with the Rollover Private Certificate when that certificate was the one that decrypted the inbound message, removing the need to configure an Alternate Local Profile on each connector during certificate rollover. * Certificate settings, such as SSL client certificates, in Application and Database connectors can now use certificates managed in Arc. Existing certificate settings continue to work. * Improved upload performance for the [Box](../../connectors/box) connector, especially when uploading to nested folders, by caching folder Ids so Arc makes fewer API calls to Box. * Updated the [Shopify](../../connectors/shopify/shopify) connector for the v26 driver, which removed the REST schema. Connections using an unsupported schema now fail with an error until switched to a supported schema. * The Catalog setting is now required for [Databricks](../../connectors/databricks/databricks) connections. Existing connections without a Catalog value must set one after upgrading. * [Odoo](../../connectors/odoo/odoo) connections now require an API key in the API Token setting. The v26 Odoo driver no longer accepts a password in this field, so existing connections that use a password must be updated to use an API key. * The [TPMC](../../tpmc/tpmc) now automatically enables Return inbound functional acknowledgements on a partner's EDI connector when a 997 or 999 Incoming Document is added, and disables it when removed, so received acknowledgements are routed to the Incoming Documents destination. * Updated the OpenSSL libraries used by the [Apache Kafka](../../connectors/apachekafka) connector to version 3.6.4 to address security vulnerabilities. ### Fixed * Fixed the [ZUGFeRD](../../connectors/zugferd) connector failing with "Unsupported version: 2.1" on .NET servers that use a comma as the decimal separator. * Fixed intermittent [SFTP](../../connectors/sftp) download failures with the error "1098 - Invalid SSH packet length" when connecting to servers that use random padding. * Fixed an issue where the certificate configured in a connection used by the [AS4](../../connectors/as4) connector was not included when exporting the connector. * Fixed an issue where retrying a failed send in the [SAP IDoc](../../connectors/sap-idoc/sap-idoc) connector re-sent IDocs from the file that had already been posted to SAP. Retries now resume after the last IDoc sent successfully. # End to End Workflow Testing Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/e2e-workflow-testing Perform end-to-end testing of TPMC partner document workflows using sample data in CData Arc. You can use the Trading Partner Management Console (TPMC) to perform end-to-end (E2E) testing for your partners and their associated workflows. Using sample data, you can test the workflow from start to finish. Testing is supported on both incoming and outgoing documents. Incoming workflow tests are further broken down into active and passive categories, as described in the following sections. To run a workflow test, click **Test Workflow** from the **Expected Documents** section of a partner's **Workflows** tab, or by clicking the ellipses next to a specific document and choosing **Test Workflow**. ![Test Workflow button on the partner Workflows tab](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi-test-workflow.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=727d0bf5ade8be749e5cea5e1d0336cf) ## Active Incoming Testing The *active* category refers to incoming workflows that receive documents by the triggering of a receive automation event from the source connector (where Arc pulls from the partner's server). Connectors in this category include [MFT](../connectors/mft-connectors) connectors such as [SFTP](../connectors/sftp), [FTP](../connectors/ftp), and [S3](../connectors/s3). Choose **Test Workflow** to trigger a test. If you use the button in the **Expected Documents** section, you are prompted to select a workflow. Once the test file has been received and processed through the workflow, a results page opens showing details about the file that was received as well as statuses for each step of the workflow. The following image shows a successful E2E test: ![Successful active incoming workflow test results](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi-active-incoming-workflow-test.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=c540ee129a9ac8c2da7d4cb24ce03d62) * Click **View Details** to open the message **Details** page of the message. * Toggle **Preview Latest Message** to see the contents of the message used in the E2E test. * By default the Arc results window only shows records that pertain to the specific partner:document relationship you chose to test the workflow on. It is possible, however, to receive documents that are not part of the specific workflow. This happens when data received from a partner's server is for a different document type, or when pulling records from a database for an outgoing test. Toggle **Show Other Records** to see these documents. ## Passive Incoming Testing The *passive* category refers to workflows that are triggered from the partner's end of the connection (the partner sends documents to you). Connectors in this category include [MFT](../connectors/mft-connectors) connectors such as [AS2](../connectors/as2), [AS4](../connectors/as4), and [OFTP](../connectors/oftp). Unlike with active workflow tests, you cannot trigger receipt of a document, so Arc provides an email template you can use to ask your partner to send some sample data. To perform an E2E test for a passive incoming document workflow, navigate to the partner's **Workflows** tab. You have two options: * Use the **Test Workflow** button in the documents section to create a sample email with details on all of the partner's configured incoming documents. Use this to send one email to your partner requesting they send sample data for all expected documents. * Hover over a workflow, click the ellipses, then click **Test Workflow** to create a sample email containing information related to one incoming document. It might look something like this: ![Sample passive incoming email template](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi-passive-email-template.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=3eecaf1dcbf17bd12b18a2c58e7311fc) ## Active Outgoing Testing Because outgoing workflows always originate from Arc, all outgoing workflows are *active*. > [!NOTE] > > The one exception to this rule is the [AMQP](../connectors/amqp) connector. It passively waits for new information to be published to the queue it is listening to. However, E2E testing works the same way as all other outgoing workflow tests. Outgoing tests involve the source connector (such as [Salesforce](../connectors/salesforce/salesforce), [MySQL](../connectors/mysql/mysql), or [Shopify](../connectors/shopify/shopify)) firing its receive automation and pulling data from a data source, database or queue. Use the same **Test Workflow** options as described above. The following image depicts a test that selects data from MySQL, translates it to X12, then sends the X12 file to the partner using SFTP. ![Active outgoing workflow test results](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi-outgoing-workflow-test.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=eeec32c93c74647243ebaae9bc702937) ## Workflow Activity Each workflow has a page that shows all of the activity (tests and real files) for that document workflow. To open it, hover over a workflow, click the ellipses, then click **Workflow Activity**. ![Workflow activity page](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi-workflow-activity.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=24fba88876ec77f9320dd3e4a1b78739) It includes the following information (click the arrow to the left of the message name to see additional details): * Latest Message * Flow Start * Flow Last Processed * Status * Processing Time Toggle **Debug Mode** to enable debug mode. Best practice is to enable debug mode for new partner:document relationships. It gives you the opportunity for manual intervention, ensuring that you only send data to your partner that you've confirmed looks correct, and that you only insert data into your database that you know is correct. Debug mode causes Arc to not automatically process messages when they are in the final step in the flow for that partner:document relationship. For incoming messages, this lets you ensure that the data you receive from your partner is what you're expecting, and that it's mapped correctly all along the flow. For outgoing messages you can ensure that the data from your database is mapped correctly, and is being translated to EDI that your partner expects. After you confirm that the contents are what you expect, click the **Send Document** button on the **Test Workflow** screen. # Managing Documents Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/edi-documents-tab Create, configure, and reuse EDI document workflows in the CData Arc TPMC Documents tab. The **Documents** tab displays a list of all configured documents, their versions, the number of partners each document is associated with, whether the document is for incoming or outgoing data, and the date it was created and last modified. You can search for specific documents, and filter by type, partner name, and direction. ![Documents tab list](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi_document_list.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=f7ff39902750c32ec22adb3524681608) To edit an existing document, click its name. You can change the document name, add partners, and update mappings. ## Creating a Document from the Documents Tab You can create new documents from both the **Partners** and **Documents** tabs. These instructions cover creating a document from the **Documents** tab. See [Select or Add Documents](./edi-partners-tab#select-or-add-documents) for information on creating documents from the **Partners** tab. From the navbar, click **EDI**, **Documents**, then click **Add Document**. ![Add Document screen](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi_document_add.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=90565e82edc4b93a534f3475c6c01edb) 1. On the **Add Document Workflow Details** screen, supply a **Name**, then use the three dropdowns to choose the **EDI Standard**, **Version**, and **Type**. 2. Specify whether this document is for incoming or outgoing data. 3. Click **Next** to go to the **Template** screen. 4. Choose whether the document workflow should be based on the built-in **Schema** for your document type, or on a sample **File** in EDI or XML format. 5. If you select **File**, click **Choose File** and navigate to the EDI or XML file to upload. 6. Click **Add Document** to save the new document. The **Document Information** section of the page that opens contains the information you supplied in the previous steps. However, before you can use the document, you need to specify which partners it is for, and what the source or destination is. The order in which these items appear in the document workflow page depends on the document direction (incoming or outgoing). The following image shows an outgoing document. ![Document workflow page](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_document_workflow.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=0fbc1fe82d246130678e1497bc0e2191) * In the **Document Source** section of an outgoing document, click **Add Source** to specify where the data is coming from. After you select and configure the source connector, the workflow mapping page opens. See [Configure the Mapping](./edi-mapping#configure-the-mapping) for detailed instruction on how to complete the mappings. * In the **Document Destination** section of an incoming document, click **Add Destination** to specify where the data flows to. After you select and configure the destination connector, the workflow mapping page opens. See [Configure the Mapping](./edi-mapping#configure-the-mapping) for detailed instruction on how to complete the mappings. * In the partners portion of the page, click **Add Partner**. Only partners that use the same EDI standard appear in the list of available partners. Use the checkboxes to specify your partners, then click **Add Partner**. ## Reusing Documents Documents are reusable between partners as often as you need, as long as the document shares the same EDI standard as that partner. Follow the steps in [Adding a Partner to an Existing Document](#adding-a-partner-to-an-existing-document) to reuse a document. ### Adding a Partner to an Existing Document You can add partners to existing documents at any time. 1. From the **Documents** tab, click a document name. 2. In the **Document Workflow** section of the page, click **Add Partner**. Only partners that use the same EDI standard appear in the list of available partners. Use the checkboxes to specify your partners, then click **Add Partner**. # Document Workflow Mappings Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/edi-mapping Configure and test EDI document workflow mappings between partner documents and source or destination connectors in CData Arc. An important component of the TPMC is configuring the mapping between the incoming or outgoing EDI document and the connector where the data is stored or sent to. *Incoming* mappings rely on receiving EDI documents from a partner over an MFT protocol, translating them from EDI to XML using an EDI standard, then mapping them to a destination connector such as a database. *Outgoing* mappings rely on pulling data as XML from a source connector (such as a database), mapping that data to an XML representation of an EDI standard, converting the XML to EDI, then sending the EDI document out to a partner over an MFT protocol. Document workflows are defined in two ways. * From the [Partners](./edi-partners-tab#select-or-add-documents) tab, click the name of a partner, then choose the **Workflows** tab. See [Select or Add Documents](./edi-partners-tab#select-or-add-documents) for instructions on how to add documents and launch a workflow. * From the [Documents](./edi-documents-tab) tab, click the name of an existing document, or choose **Add Document** to create a new one. See [Creating a Document from the Documents Tab](./edi-documents-tab#creating-a-document-from-the-documents-tab) for instructions on how to add a document and launch a workflow. ## Add Document Workflow Defining document workflows includes the following steps: 1. Complete the **Add Document Workflow** screen. 1. When you add a document from the **Partners** tab, click **Add Workflow** to launch the document workflow. In this case, the fields on the **Add Document Workflow** screen are prefilled. You can change the document name, but all other fields are fixed. 2. When you add a document from the **Documents** tab, you must manually complete the fields on the **Add Document Workflow** screen. 2. [Select and configure the source or destination connector](#select-and-configure-the-source-or-destination) 3. [Configure the mapping](#configure-the-mapping). Mapping is applicable for all connectors except [File](../connectors/file), [AMQP](../connectors/amqp), and [Apache Kafka](../connectors/apachekafka) which do not use mapping. See [Document Workflow Without Mapping](./workflow-nomapping) for more information. ### Select and Configure the Source or Destination When you create an *incoming* document, you must specify the destination connector where the data will be added. When you create an *outgoing* document, you must specify the source connector to pull the data from. The available connectors are the Arc [Database](../connectors/database-connectors) and [Other](../connectors/other-connectors) connectors, the [File](../connectors/file) connector, the [Workspace Send](../connectors/workspace-send) connector (for incoming documents only), and the [Workspace Receive](../connectors/workspace-receive) connector (for outgoing documents only). The following images show configuring a source for an outgoing document, but the same principles apply for incoming document destinations. In this example, the source is being defined from a partner's **Workflow** tab, but you can achieve the same result from the **Documents** tab. 1. From the **Workflow** tab, click **Add Source**. ![Add Source button on the Workflow tab](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi_add_source.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=395ec9f087fa9031cc31314e341f723e) 2. Highlight a connector in the **Select Source** list and click **Add**. ![Select Source connector dialog](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_select_source_connector.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=1982fade392cae65c5c1dcacd7f6f129) 3. Use the plus sign to create a new **Connection**, or use the dropdown to select an existing one. ![Source connector configuration](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_source_config.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=da470c1fef607be15b331db8a372fcf0) 4. When you add a new connection, complete the following fields in the **Add Connection** window: * Name * Server * Port * User * Password * AWS Access Key (optional) * AWS Secret Key (optional) * AWS Role ARN (optional) 5. Click **Test Connection**. 6. The only action available for an outgoing document is `Select`, so when you have added and tested your connection, click **Next**. (The action for an incoming document is `Upsert`. See [Actions](#actions) for more information.) This opens the **Action** connector configuration tab, where you complete the mappings. See [Configure the Mapping](#configure-the-mapping) for instructions on how to use this tab. ![Mapping start page showing Action tab](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_mapping_source_structure.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=96f852a94d3f01e8492ae749708f430e) > [!NOTE] > > * Use the other configuration tabs (such as **Settings** and **Automation**) to complete the connector configuration. For details on the configuration options, click the question mark help icon to open the connector documentation. > * If you need to modify the connection after the original setup, use the **Connection** field on the **Settings** tab. ## Configure the Mapping Once you have established the connection, you must define the structure of the data you are mapping. If you're creating an incoming document, click **Add Destination Structure**. If you're creating an outgoing document, click **Add Source Structure**. For outgoing documents, the source structure editor lets you choose between **Parent and Child Tables** and **Joined Tables** structure types, configure which columns to include, and apply filters. See [Configuring the Source Structure](./edi-source-structure) for details. The following image of a completed mapping has been annotated to explain each element on the screen. Each number corresponds to a description below. ![Annotated outgoing document mapping](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_mapping_outgoing_annotated.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=c9154228c5fe2ed7c57e7512c1473bd7) 1. The document or connector name. 2. The connector type. 3. The action for the document. Values are **Upsert** for incoming documents, and **Select** for outgoing documents. 4. Test status of the mapping. Values are **Test Successful**, **Tested with Errors** or **Not Tested**. 5. Toggles XML streaming on or off. When on, the mapping engine drops the Document Object Model (DOM) for the current iteration of the top-most `Foreach` in the mapping after completing the loop. This can significantly improve performance for large XML documents. In situations where the mapping engine cannot resolve the mapping, it automatically toggles XML streaming off. 6. Toggles between regular and compact views of the mapping (the compact view can be helpful with large template files). 7. Toggles whether to show or hide the XML attributes. 8. Refreshes the mapping. 9. Launches [AI-Assisted Mapping](../mapping/mapping-config#ai-assisted-mappings). 10. Opens the **Test Mapping** page. The test result is shown in #4. (See [Testing Mappings](#test-mappings) for more information.) 11. Saves the mapping. 12. The target table in the configured data source. 13. The direction of the mapping. 14. The schema of the EDI document (an 810 invoice in this example). 15. Search bars on the source and destination sides. 16. Structure of the source table. 17. The name of the table. This is also known as a table node. 18. A column in the table. This is also known as a column node. 19. The structure of the destination document. 20. Completed mappings. The mapping page enables you to create the links between the EDI document and the source or destination connector. For incoming documents, the EDI document structure is in the left pane and the connector table structure is in the right pane. For outgoing documents it's the reverse, as you can see in the annotated image above. See [Configure the Mapping](../mapping/mapping-config) in the [Mapping](../mapping) documentation for detailed instructions on how to use the mapping editor. > [!NOTE] > > To edit a previously saved mapping, click **EDI > Documents**, then click the document name. In the **Document Workflow** section, click the link to the connector in the **Document Destination** (for incoming documents) or the **Document Source** (for outgoing documents). ### Understanding Source and Destination As mentioned previously, when you create an *incoming* document, you must specify the destination connector where the data will be sent. The EDI document structure is in the left pane and the connector table structure is in the right pane. When you create an *outgoing* document, you must specify the source connector to pull the data from. Here, the connector table structure is in the left pane and the EDI document structure is in the right pane. Arc provides a number of ways in which you can manipulate the information in the table and document structures. See [Table Structure Nodes](#table-structure-nodes) and [Document Nodes](#document-nodes) for details. To change the document schema, or to upload a template file that represents the source or destination XML structure, click the link to the document schema and choose **Upload File**. This is helpful if your partner provides you with a sample document of what they are expecting to send and/or receive. For example, if your partner provides you with a sample X12 850, you can convert it into XML by running it through an [X12 connector](../connectors/x12) in `X12 to XML` mode in a regular workspace. Then you can upload the output XML as a template file. ![Upload Template File dialog](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_upload_template_file.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=78bad1a06667b6d6bb08f3b54cf5d231) Browse to the template file, then click **Upload Template File**. #### Actions Two actions are available when you configure a connector: * Upsert is the action for incoming documents. This action inserts or updates data into the target destination table. If a record already exists in the destination table, an update is performed on the existing data using the values provided from the input. * Select is the action for outgoing documents. This action retrieves data from the target data source. ### Table Structure Nodes This section describes the options available in the mapping pane that displays the source or destination table structure. There are two types of nodes in the structure: table nodes and column nodes. Table nodes represent the table itself and column nodes represent the columns in the table. #### Table Nodes Table nodes are indicated with a table icon. When you hover over a table node, you have two additional options: * Click the funnel icon to open the **Add Mapping Conditions** page which enables you to add filter conditions to the mapping. See [Using the Condition Editor](../mapping/mapping-condition-editor) for more information. (Only available for incoming document mappings.) * Click the ellipses to open a list of options for that table node: * **Rename Table** lets you change the name of the node. * **Remove Table** removes the node. * **Table Settings** opens an edit window that lets you select which columns to include in the mapping, and whether to apply [filters](./edi-source-structure#filters). (Only available for outgoing document mappings.) * **Edit Columns** opens an edit window that lets you select which columns to include in the mapping. (Only available for incoming document mappings.) * **Add** opens a new list of nodes that you can create and use in your mappings. (Only available for incoming document mappings.) The options are: * **Add Header** creates a header node that you can map a value to. The mapped value is not included in the output file but is added as a header on the mapping output message. * **Add Loop** creates a loop node above the selected node and that node becomes nested within the loop node. See [Mapping Loops](../mapping/mapping-config#mapping-loops) for more information. * **Add Condition** lets you add conditions to destination nodes so that data is only upserted to the destination if the condition is true. See [Using the Condition Editor](../mapping/mapping-condition-editor) for details. * **Add Code Script** opens the **Add Script** window where you can name the script node and supply custom ArcScript. * **Add Variable** creates a variable node below the selected node. See [Variables](../mapping/mapping-node-value-editor#variables) for more information. #### Column Nodes Column nodes represent columns in a table. Every column node has a type, which is the column's datatype in the database. There are seven different datatypes, each one with its own symbol. Hover over a column to see its datatype, size, and whether it is nullable. The datatypes are: * String * Number * Binary * Boolean * Date * Time * Date and Time Hover over a column row and click the ellipses to open a list of options for that node: * **Rename Column** lets you change the name of the node. * **Remove Column** removes the node. * **Add** opens a new list of nodes that you can create and use in your mappings. (Only available for incoming document mappings.) The options are: * **Add Header** creates a header node that you can map a value to. The mapped value is not included in the output file but is added as a header on the mapping output message. * **Add Loop** creates a loop node above the selected node and that node becomes nested within the loop node. See [Mapping Loops](../mapping/mapping-config#mapping-loops) for more information. * **Add Code Script** opens the **Add Script** window where you can name the script node and supply custom ArcScript. * **Add Variable** creates a variable node below the selected node. See [Variables](../mapping/mapping-node-value-editor#variables) for more information. * **Add Tracking** adds [tracked headers](../getting-started/administration/activity#tracked-headers) to the mapping. When a node has tracking enabled, it displays a compass icon. ### Document Nodes Document nodes refer to the nodes on other side of the mapping: the non-database side. These nodes are not explicitly related to a data source (a table with columns) but rather represent the EDI data in an XML structure. These nodes are mapped to for outgoing documents and mapped from for incoming documents. For outgoing documents (using the Select action), click the funnel icon to open the **Add Mapping Conditions** page which enables you to add filter conditions to the mapping. See [Using the Condition Editor](../mapping/mapping-condition-editor) for more information. Click the ellipses to use the following options in the destination document: * **Rename Node** lets you change the name of the node. * **Edit XPath** lets you edit the node XPath (only on elements already mapped as `Foreach`). * **Delete Node** lets you delete the node from the document. * **Add Node** * **Add Sibling** adds a node at the same level as the selected node. * **Add Attribute** adds an attribute to the selected node. * **Add Child** adds a node as a child of the selected node. * **Add Header** creates a header node that you can map a value to. The mapped value is not included in the output file but is added as a header on the mapping output message. * **Add Loop** creates a loop node above the selected node and that node becomes nested within the loop node. See [Mapping Loops](../mapping/mapping-config#mapping-loops) for more information. * **Add Condition** opens the [Condition Editor](../mapping/mapping-condition-editor) where you can add conditions so that data is only upserted to the destination if the condition is true. * **Add Code Script** opens the **Add Script** window where you can name the script node and supply custom ArcScript. * **Add Variable** creates a variable node below the selected node. See [Variables](../mapping/mapping-node-value-editor#variables) for more information. * **Cut Node** cuts the selected node from its current location. * **Copy Node** copies the selected node. * **Paste as Child** pastes the cut or copied node as a child of the selected node. For incoming documents (using the Upsert action), click the ellipses in the source document structure to use the following options: * **Rename Node** lets you change the name of the node. * **Delete Node** lets you delete the node from the document. * **Add Node** * **Add Sibling** adds a node at the same level as the selected node. * **Add Attribute** adds an attribute to the selected node. * **Add Child** adds a node as a child of the selected node. * **Cut Node** cuts the selected node from its current location. * **Copy Node** copies the selected node. * **Paste as Child** pastes the cut or copied node as a child of the selected node. You can also use **Edit Node** to open the **Edit Node Value** page. See [Using the Node Value Editor](../mapping/mapping-node-value-editor). ## Test Mappings Once you have configured the mapping, you should test it. Click the **Test Mapping** button at the top of the page. The following image shows testing an incoming document. ![Test Mapping page](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_test_mapping.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=7560ecd5442cf0514353c2e7cf0c6bef) The source side of the test page has two tabs: **XML** and **Headers**. The **XML** tab is populated with the XML representation of the document or table structure to test, but you can edit it as necessary. The **Headers** tab is visible only if your mapping uses [Tracked Headers](../getting-started/administration/activity#tracked-headers); use it to add any header values you want to use in the mapping. When you are ready, click **Test Mapping**. Test result indicators appear on the partner **Workflow** page and on the mapping itself. The values on the mapping are **Test Successful**, **Tested with Errors** or **Not Tested**. The **Workflow** page shows a check for successful tests, a brown exclamation mark for untested mappings, and a red exclamation for tests with errors. Hover over the icons for details about the errors. > [!NOTE] > > When you test incoming documents, you are testing with live data. When you click **Test Mapping**, Arc tries to perform the `upsert` after it tests the mapping. ## Filters and Query View For information on filtering the data the connector selects, including creating filters, filter groups, and using Query View, see [Filters](./edi-source-structure#filters) in the Configuring the Source Structure topic. # Managing Partners Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/edi-partners-tab Configure trading partner workflows including MFT connectors, EDI translation, and document assignments in the CData Arc TPMC. The **Partners** tab contains a list of all trading partners you have defined, the file transfer protocol used by that partner, the EDI standard, the number of documents and a count of incoming and outgoing messages. ![Partners tab list](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_partner_list.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=48d6aa6cd7af37f8c57d4b7af9921ff7) Click **Add Partner** to configure a new partner. Supply a **Partner Name** and click **Add Partner** to be taken to the new partner's **Workflows** tab. ## Configure a Partner Workflow The following image of a fully configured partner workflow has been annotated to explain each element in the workflow. Each number corresponds to a description below. ![Annotated partner workflow](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_partner_workflow_annotated.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=765ede302ac9b43e7fa6048944716ebd) 1. The [MFT](../connectors/mft-connectors) or [File](../connectors/file) connector that handles incoming and outgoing file transfers. 2. The [EDI](../connectors/edi-connectors) connector that translates documents to and from XML. 3. Arrows indicating the direction of data flow. Two arrows indicate that the flow is bidirectional. 4. The defined document that you exchange with your partner. Clicking the document name takes you to that document's workflow in the **Documents** tab. 5. The table in the application connector where data is upserted to or selected from. The STATUS column shows whether the mapping has been tested. The checkmark indicates test success, and the exclamation mark indicates that the mapping hasn't been tested or that an error has occurred during testing. 6. Status of all the document workflow definitions. For example, if a workflow for a document is incomplete, an orange **x of y Incomplete** badge appears in place of the **Workflows Complete** badge. 7. Count of incoming and outgoing messages processed. ### Configure File Transfer Protocol When you add a new partner, you need to configure the connector to use for both incoming and outgoing file transfers. Click **Add Connector**, then select from the list of available connectors. When you click **Add**, the connector configuration pane opens. For details on the configuration options, click the question mark icon to open the documentation for your MFT connector. When you are finished, click **Save**. > [!NOTE] > > If you need to change the MFT connector type later, you must delete the existing connector, then add a new one. However, you can edit the current configuration options at any time by clicking the tablet and pencil icon. ### Configure EDI Translation The next step is to configure the connector to use to translate partner documents into XML so they can be used in Arc. Click **Add Connector**, then select from the list of available EDI connectors. When you click **Add**, the connector configuration pane opens. For details on the configuration options, click the question mark icon to open the documentation for your EDI connector. When you are finished, click **Save**. > [!NOTE] > > If you need to change the EDI connector type later, you must delete the existing connector, then add a new one. However, you can edit the current configuration options at any time by clicking the tablet and pencil icon. If you change the EDI version of the connector, you are notified that all documents configured for the current partner will be removed. ### Select or Add Documents Once you have configured the MFT and EDI connectors, you can add documents to the workflow. Use the **Add Existing** or **Add New** buttons to continue. * Choose **Add Existing** to select from existing documents that use the same EDI connector for translation. Use the checkboxes to choose your document(s), then click **Add Documents**. * Choose **Add New** to create a new document on the fly. Choose the schema version for your EDI document, use the checkboxes to select from the list of available documents, then click **Add Documents**. The new documents appear in the list of incoming or outgoing documents with a status of **Incomplete**. Use the **Add Workflow** link to add a workflow for each one. See [Add Document Workflow](./edi-mapping#add-document-workflow) for detailed instructions. > [!NOTE] > > You can also add documents from the **Documents** tab. See [Managing Documents](./edi-documents-tab) for more information. ### Add Workflow When you add documents from the partner **Workflows** tab and click the **Add Workflow** link, Arc prefills the document workflow information on the **Add Document Workflow** screen based on the MFT and EDI settings for the partner. You can change the **Name**, but all other fields are fixed. See [Add Document Workflow](./edi-mapping#add-document-workflow) for detailed instructions on defining the workflow. # Configuring the Source Structure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/edi-source-structure Define the source structure for outgoing EDI document workflows in CData Arc. For outgoing document workflows, you must define a source structure that tells the connector which data to retrieve from your data source. The source structure is configured in the **Edit Source Structure** dialog, which you open by clicking **Add Source Structure** on the **Action** tab. ![Add Source Structure button on the Action tab](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_mapping_source_structure.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=96f852a94d3f01e8492ae749708f430e) Once you define the source structure, the source columns appear in the left pane of the mapping editor, ready to be mapped to the destination EDI document structure. ![Mapping view after source structure is defined, showing source columns alongside the destination EDI document](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_mapping_tables_defined.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=0d11de156440e63137d321df8c4f8971) ## Structure Type When you open the **Edit Source Structure** dialog, choose a **Structure Type** from the dropdown. The structure type determines how the connector queries your data source. * **Parent and Child Tables**: Queries tables using a parent-child hierarchy. You can add additional tables as child tables. This is described in [Parent and Child Tables](#parent-and-child-tables). * **Joined Tables**: Issues a single JOIN query that combines data from two or more related tables. This is described in [Joined Tables](#joined-tables). > [!NOTE] > > Joined Tables issues a single query to your data source, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ## Parent and Child Tables When you select **Parent and Child Tables** as the structure type, use the dropdown to select a target table from your data source. Use the arrow to add child tables to your structure. When you have finished with your selections, click **Apply**. ### Results After you click **Apply**, an **Items** node appears in the source pane of the mapping editor. To configure the columns and filters for your query, click the ellipsis on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. In the **Columns** tab, all columns from the selected table are listed. By default, all columns are selected. Uncheck a column to exclude it from the query output. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. ### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. * **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. * **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed. ## Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in your data source relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_join_designer.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=2615628cc1da9e59d286875f621b9da1) Each row in the JOIN designer represents one join relationship and contains the following fields: * **Join Type**: The type of SQL join to perform. The available options depend on what your data source supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**. * **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query. * **Left Column**: The column from the left table to join on. * **Right Table**: The related table on the right side of the join. * **Right Column**: The column from the right table that matches the left column. The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_join_addtable.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=e40ceaef89efc903a1c66477f71520ee) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. ### Results After you click **Apply**, a **Results** node appears under the **Items** node in the source pane of the mapping editor. To configure the columns and filters for your join output, click the ellipsis on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_join_results.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=f7e00ad612c9179bee3994a20e760cc6) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. ### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. * **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting. * **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed. You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. In the settings dialog, click the **Filters** tab to configure filters. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps: 1. Click **Add Filter**. 2. In the column dropdown, choose the column that you want to filter. 3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**. 4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown. The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_filter.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=0692db48d23814663970c914e605c8fa) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top: * **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000. * **And / Or** toggle: Determines how this group relates to adjacent filters or groups. * If you select **And**, this group's condition must be true in addition to adjacent conditions. * If you select **Or**, this group's condition is an alternative to adjacent conditions. The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_filter_group.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=0750682557fd8547feeb26d3563220c1) In this example, the filter condition is met when all of the following are true: * The **Vendor** column contains **CData** * The **Status** column equals **Active** or **Pending** * The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000) If this filter condition is met, the connector selects the matching data and passes it into the mapping. ### Using Query View Toggle on **Query View** to write a custom SQL query that selects data from your data source. This overrides any queries created in the designer. ![Query View mode showing a custom SQL query](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_query_view_263.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=d2ff9332a4a405ce95e13a5368497b59) # Partner Dashboard Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/partner-dashboard View key metrics and recent messages for a trading partner in the CData Arc TPMC. Each trading partner has its own dashboard, which is the EDI **Partners** landing page. ![Partner dashboard key metrics](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_partner_dashboard.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=643b8874ec09efce7542c2b693f2a812) By default, it displays metrics for the last 7 days, but you can use the calendar dropdown to select your preferred time period. The options are: * Today * Past 24 hours * Past 3 days * Past 7 days * Past 30 days * Past 3 months * Past 12 months > [!NOTE] > > The time periods shown vary based on your [cleanup interval settings](../getting-started/administration/settings/cleanup-settings), so you might not see all of the time periods listed above. The dashboard is divided into a number of sections, each of which is covered below. The dashboard is user-specific, so you can customize it to fit your needs by using the **Edit** button near the top of the page, or the **Edit** links on certain cards. ## Trends The Trends section displays a stacked bar chart of daily message counts over your selected time period, with successful messages in dark blue and errors in red. ## Key Metrics You can display between two and eight metrics that provide a view into your TPMC partner's health, volume, and usage. To pick your metrics, click the **Edit** button, then click the **Edit** link in the Key Metrics section. Choose from the following list: * Total Messages * Total Transactions * Total Transactions Sent * Total Transactions Received * Total MFT Transactions * Total EDI Messages * Total Errors * Total Send Errors * Total Receive Errors * Total Connectors * Active Connectors * Connectors in Backoff * Total Workspaces * Active Workspaces * Total Bytes Processed * Average Bytes Processed * Average Processing Time * Success Rate * Failure Rate * SLA Violations * At Risk SLAs ## Recent Messages ![Partner dashboard recent messages](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi_dashboard_messages.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=a702da2c70db405293b9434590b0b663) This is a simplified view of what you see on the **Activity > Messages** tab. It displays the 25 most recent messages. Click the **See all** link to go to the Messages tab and view the full list. See [Messages Tab](../getting-started/administration/activity#messages-tab) for details. # Partner User Access Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/partner-user-access View the list of users with access to the CData Arc TPMC and their roles. The **User Access** tab is a read-only list of all users with access to the TPMC. It displays each user's name, roles, and current status. You can filter the list by role and by status. See [Users](../getting-started/administration/settings/users) and [User Roles](../getting-started/administration/settings/user-roles) for more information. # EDI Trading Partner Management Console Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/tpmc Overview of the CData Arc Trading Partner Management Console (TPMC) for configuring EDI partner relationships and document workflows. The Trading Partner Management Console (TPMC) is a central place where users can go and configure an EDI partner relationship, map any necessary documents, and have that partner's details as well as the document mapping be reusable. To access the TPMC, click **EDI** on the navbar. The EDI tab has two sub-tabs where you define your [Partners](./edi-partners-tab) and your [Documents](./edi-documents-tab). ![EDI TPMC landing page](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_landing_page.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=535ed829fe5e7531228b376be7188609) At a high level, the TPMC has four main components: * The trading partner * The document source * The mapping (optional) * The document destination For example, if you have a trading partner relationship with Amazon and you need to trade documents back and forth, the TPMC makes it easy to create the relationship once and reuse it for an unlimited number of mappings in the future, without you having to re-enter all of the trading partner's details. You can configure four types of documents in the TPMC: * Incoming EDI documents with mapping * Incoming EDI documents without mapping * Outgoing EDI documents with mapping * Outgoing EDI documents without mapping Using Amazon and an EDI 810 invoice document (outgoing EDI with mapping) as an example, the process follows these steps: 1. Create a partner for Amazon. 2. Configure the outgoing MFT protocol (such as AS2). 3. Select the EDI standard to translate the database information to XML (X12, EDIFACT, and so on). 4. Select the source of the information to populate the 810 invoice document (for example, a MySQL database). 5. Configure the document mapping (for example, MySQL output to XML to X12). All stages of this configuration are saved and can be reused later. > [!NOTE] > > Configurations and mappings on the EDI tab are completely separate from what is available on the [Flows](../flows/flows) page. In addition, mappings and partner configurations configured on the Flows page are not available on the EDI tab. However, TPMC partners cannot have the same name as a Flows [workspace](../flows/workspaces). # Document Workflow Without Mapping Source: https://docs.arc.cdata.com/26.4/self-hosted/en/tpmc/workflow-nomapping Configure incoming and outgoing TPMC document workflows that do not require a mapping step in CData Arc. You can define incoming and outgoing workflows without a document mapping in the TPMC. For example, you might have a workflow where the document destination is a local disk. When you choose the destination connector, choose the [File connector](../connectors/file). See [Select and Configure the Source or Destination Connector](./edi-mapping#select-and-configure-the-source-or-destination) for detailed instructions. Because there is no mapping component for connectors that use unstructured information, there is no **Action** tab in the connector configuration pane. When you choose the connector, you are taken directly to the **Settings** tab. ![Connector settings without a mapping tab](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edi_no_mapping.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=0e71d947032645e50d93593ae10b0d8d) The following connectors do not use mapping: * [AMQP](../connectors/amqp) * [Apache Kafka](../connectors/apachekafka) * [File](../connectors/file) # AS2 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/as2 Configuration and usage guide for the CData Arc AS2 connector. The AS2 connector supports sending and receiving messages using the Applicability Statement 2 (AS2) protocol. ## Key Capabilities * Secure and reliable B2B document exchange * Full security support including signing, encryption, and compression * Synchronous and asynchronous MDN receipt handling with automatic retry and resend * Support for very large messages with streaming and restart capabilities * Comprehensive certificate management for trading partner relationships * Advanced features like AS2 reliability and FDA extension parsing * Certified by the [Drummond Group](https://www.drummondgroup.com/certified-products/b2b-interoperability/) ## Overview An AS2 connection is configured in two places. Configure the AS2 [Profiles](../getting-started/administration/profiles) page with a local AS2 identifier, private certificates, and other information that is global across all AS2 connections. Then configure individual AS2 connectors with connection settings specific to a single trading partner. When an input file is processed by an AS2 connector, it is packaged and sent to the specified trading partner. When Arc receives a file over AS2, it attempts to route the file to a specific AS2 connector. The application uses the AS2 identifiers in the AS2 message to determine which AS2 connector should receive the file. When a file is routed to an AS2 connector, it is placed in the connector's **Transactions** tab, or is passed along to the next connector in the flow. AS2 connectors support all security and reliability mechanisms that make AS2 a popular protocol. See an [explanation of the AS2 protocol exchange](#what-happens-in-an-as2-exchange) and an [explanation of the certificates used for AS2 security](#certificates) for more information. ### Video Resources Watch this short video to learn how to quickly set up an AS2 connector. This is part one of a three-part video series that covers each step of an end-to-end B2B integration: managed file transfer (using AS2 as an example), back-end integration, and EDI translation and mapping. Links to the other parts of the series are below. [https://www.youtube.com/embed/YjMpP70fczo](https://www.youtube.com/embed/YjMpP70fczo) * Part Two: [Back-End Integration in CData Arc—Applications and Databases](https://youtu.be/uhWYwsXRc3Q) * Part Three: [EDI Mapping and Integration in CData Arc](https://youtu.be/wk92g8_Jmn0) ## Profile Configuration The AS2 Profile must be configured before connections can be established with individual AS2 connectors. Click **Profiles** on the navbar. ### AS2 Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **AS2 Identifier** | Your AS2 identifier. Messages sent by Arc include this value as the `AS2-From` header. Incoming messages must have this value as the `AS2-To` header to be successfully received. AS2 Identifiers are case-sensitive. | #### Personal Certificate *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in an AS2 transaction: a corresponding public key is also generated with the same filename and a `.cer` extension. | | **Certificate Password** | The password required to access the **Private Certificate**. | | **Rollover Private Certificate** | A secondary private certificate that can be used to decrypt incoming messages if decryption with the **Private Certificate** fails. Only use this setting when transitioning certificates when an overlap period is necessary to accept incoming messages encrypted with either the old or the new certificate. The rollover certificate is never used to sign outgoing messages, which can cause signature verification issues when sending to trading partners that are still using the old certificate. | | **Rollover Certificate Password** | The password required to access the **Rollover Private Certificate**. | #### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Asynchronous MDN URL** | The return address that Arc sends to trading partners when requesting asynchronous MDN receipts. Arc populates this field from the **Base URL** and the default MDN endpoint, `ReceiveMDN.rsb`, when the AS2 profile is first configured. If the server address or port changes, open the AS2 profile and save it to update this field. | | **Receiving URL** | The URL at which Arc listens for incoming AS2 messages. Share this URL with all your trading partners. | | **Publish AS2 profile settings** | If enabled, an endpoint is exposed where trading partners can view AS2 configuration details including identifiers, URLs, algorithms, and certificates. | | **Public URL** | The endpoint at which trading partners can view AS2 configuration details. This URL can be shared with trading partners to simplify the communication of AS2 specifics. | | **Public Certificate** | The public key certificate to be shared on the public configuration page. Set this to the encryption certificate that trading partners should use when sending AS2 messages to Arc. This certificate should have the same name as the **Private Certificate** but with a `.cer` extension. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration Once you configure the global AS2 profile settings, create and configure individual AS2 connectors for each trading partner on the [Flows](../flows/flows) page. ### Settings Tab #### Trading Partner Info *Settings for identifying and connecting to a specific AS2 trading partner.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **AS2 Identifier** | The AS2 identifier specific to the target trading partner. This value is included in the AS2 headers for outgoing messages, and it is also used to route incoming AS2 messages to the appropriate AS2 connector. | | **Partner URL** | The trading partner's public endpoint where outgoing AS2 messages should be sent. | #### Connection Info *Settings related to connection parameters for the specified trading partner.* | Setting | Description | | - | - | | **Send Message Security** | Whether to sign and/or encrypt outgoing AS2 messages. CData strongly recommends you use signatures and encryption. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming AS2 messages. An error is thrown if a received message does not have a required security parameter. | | **Compression** | Whether to compress the payload of outgoing messages. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Encryption Algorithm** | Which encryption algorithm to use if you require message encryption. | #### MDN Receipts *Settings related to requesting MDNs when sending AS2 messages.* | Setting | Description | | - | - | | **Request MDN Receipt** | Whether an MDN receipt should be returned in response to outgoing AS2 messages. CData strongly recommends that you request MDN receipts. | | **Security** | Whether the MDN receipt should include a signature block verifying the message integrity and identity of the recipient. CData strongly recommends you use this option. | | **Delivery** | Whether the MDN should be returned as a direct response to the outgoing AS2 message (Synchronous) or returned later as part of a separate connection (Asynchronous). Synchronous MDNs are recommended unless the size of the AS2 message is very large (50MB is a common threshold), in which case processing the message and delivering a synchronous MDN might strain the connection timeout duration. | #### Trading Partner Certificates *Settings related to the public key certificates provided by the trading partner.* | Setting | Description | | - | - | | **Encryption Certificate** | The public key certificate used for AS2 encryption when sending messages. This certificate must be paired with the trading partner's private certificate, and the trading partner should provide a public key certificate when sharing AS2 configuration details. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the partner's AS2 system requires HTTPS instead of HTTP. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or it can be set to `Any Certificate` to unconditionally trust the target server's identity. | #### Public Profile The AS2 profile configuration details published on a public endpoint accessible to trading partners. This endpoint is configured on the [Profile](../getting-started/administration/profiles) page. ### Advanced Tab #### Very Large Message Support (VLM) *Settings used to support sending large AS2 messages.* | Setting | Description | | - | - | | **Streaming** | Whether to use HTTP Chunked Transfer Encoding when sending messages. This allows the application to send portions (chunks) of the message sequentially to avoid overloading the connection. | | **AS2 Restart** | Whether to support resuming transmissions that were interrupted. This is useful when streaming large messages in chunks. | > [!NOTE] > > Not all AS2 systems support this feature. #### Reliability *Settings related to the Reliability feature of the AS2 protocol.* | Setting | Description | | - | - | | **AS2 Reliability** | Whether to reuse AS2 Message IDs when resending a document. This helps prevent the same document from being processed twice on the receiving side. | | **AS2 Reliability Interval** | The number of days that the application remembers files that have been received, even if they have been removed from the **Transactions** tab. If you set a value of 0, the application never clears the received file list. | #### Alternate Local Profile *Settings that override the AS2 configuration on the **Profiles** page for this specific AS2 connector. Setting an alternate local profile lets you use different local certificates and identifiers for certain trading partners.* | Setting | Description | | - | - | | **Local AS2 Identifier** | Your AS2 identifier. Overrides the **AS2 Identifier** on the Profiles page. | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Overrides the **Private Certificate** on the Profiles page. | | **Certificate Password** | The password required to access the local private certificate. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Use Profile Settings** | Whether to use the **Private Certificate** configured on the Profiles page as the TLS certificate for client authentication. | | **Private Certificate** | The private certificate presented during TLS client authentication. Only applicable if you are not using the same private certificate from the Profiles page. | | **Certificate Password** | The password required to access the TLS client certificate. | #### HTTP Authentication *Settings related to HTTP client authentication.* | Setting | Description | | - | - | | **HTTP Authentication** | Whether to use client HTTP authentication. | | **HTTP Authentication Type** | Whether to provide HTTP authentication credentials in an encrypted format (Digest) or in plain text (Basic). Only use Basic authentication if the connection is an HTTPS connection (instead of HTTP). | | **User** | The user credential for HTTP client authentication. | | **Password** | The password credential for HTTP client authentication. | #### Custom Headers A set of custom headers to include in the outgoing message. #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Async MDN Timeout** | The HTTP response timeout (in seconds) to observe when returning an asynchronous MDN receipt. The default is 60 seconds. | | **Duplicate File Action** | How the connector should behave when it receives an AS2 message with a filename that the connector has seen before (a duplicate). When set to `Continue`, duplicate filenames are renamed, but no warning is sent to the trading partner. When set to `Warning`, the connector renames duplicate filenames and returns a warning in the MDN. When set to `Warning - Ignore File`, duplicate filenames are not renamed but the connector returns a warning in the MDN. When set to `Failure`, the connector does not accept the file and returns an error in the MDN. The connector remembers filenames that are received for the duration set in **Duplicate File Interval**. | | **Duplicate File Interval** | The length of time (in minutes) that a file with the same filename is considered a duplicate. In other words, the length of time the connector remembers that a specific filename has already been received. If set to 0, the filenames are stored indefinitely. | | **Extension Map** | A set of name-value pairs that maps file extensions to the desired HTTP Content-Type header value. By default, the application maps the following file extensions to content types: `.xml` → `application/xml`, `.edi` or `.x12` → `application/edi-x12`, `.edifact` → `application/edifact`. All other file extensions are sent with an `application/octet-stream` content type. To add or overwrite mappings, supply a comma-delimited list in `extension=contenttype` syntax (for example, `.txt=text/plain,.edi=application/edifact`). | | **HTTP Subject** | The HTTP Subject header to include in the outgoing AS2 message. This header is not used in the AS2 protocol, but can be used by some solutions for additional business logic processing. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Message Id** | Supply a value to replace the suffix of the AS2 Message Id on the outbound message (the portion after the @). For example, `AS2_Test-20231206-091640691-IcJq@MyValue`. | | **Parse FDA Extensions** | Whether to parse outgoing filenames to include FDA-specific headers in the AS2 message. If enabled, files on the **Transactions** tab should have the FDA center and the FDA submission type as the first two parts of the filename, separated by periods (for example, `CDRH.eMDR.myfile.txt`). The application automatically translates these filename prefixes into the appropriate FDA-required headers. | | **Partner Signing Certificate** | If the trading partner uses separate private certificates to sign messages and to decrypt messages, set this to the public key certificate that corresponds to the partner's signing certificate. The trading partners should be able to provide this public key certificate. | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. The same algorithm is requested for the corresponding MDN receipts. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Temp Receive Directory** | If set, the application writes received files to the temporary directory as they are received, then moves the finished files to the **Transactions** tab. This ensures that partial files are never present on the **Transactions** tab, even when the connector receives very large files. | | **HTTP Headers** | To include headers from HTTP messages as metadata on the downloaded messages, provide the headers as a comma-delimited list. | | **Log MDNs** | Whether the connector keeps a copy of each MDN receipt in the `.mdn` log for the transaction. When checked (the default), MDN receipts are retained regardless of the **Log Level**, including `None`. When cleared, no MDN receipt is written at any **Log Level**. Because MDNs serve as receipts of delivery, keep this checked if you might need to prove delivery to a trading partner. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
> [!NOTE] > > For AS2 connectors, the `.mdn` log is written regardless of the **Log Level**, including `None`, as long as **Log MDNs** is enabled on the **Advanced** tab. Lowering the **Log Level** does not discard MDN receipts. #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as AS2 messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connector retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous MDN receipt is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Trading partners must provide some of the connection details that are required when you configure a new AS2 connector. At a minimum, these details should include: * AS2 Identifier * Partner URL * Partner Certificates ### AS2 Identifier Your trading partner is identified by their AS2 identifier in an AS2 transaction. When sending outgoing requests, the AS2 identifier is used in the header of the request to indicate the recipient. To establish an AS2 self-test, the identifier should be set to the same value as the **AS2 Identifier** on the [Profiles](../getting-started/administration/profiles) page. > [!NOTE] > > This value is case-sensitive. ### Partner URL The Partner URL is the endpoint where the trading partner receives AS2 transmissions. Outgoing AS2 messages are sent to this target endpoint, which must be unique for each trading partner. You can test the Partner URL with a web browser to check for networking or connectivity issues. To establish an AS2 self-test, the target URL should be identical or nearly identical to the **Receiving URL** on the [Profiles](../getting-started/administration/profiles) page. You can replace the domain name from the Profiles page with the loopback address localhost to keep the AS2 transaction in the local network. An example local self-test URL is `http://localhost:8001/pub/Receive.rsb`. If you do not replace the domain name with localhost, the AS2 message is routed outside of the local network. You can use this to check network configuration settings and to make sure that the message can reach Arc through any firewalls. Trading partners might sometimes provide more than one URL: a Receiving URL and a URL for asynchronous MDNs. In this case, only the Receiving URL (Partner URL) needs to be configured; the application can read the asynchronous MDN URL from incoming AS2 transmissions. ### Partner Certificates Each AS2 connector must be configured with the public key certificate(s) for the target trading partner. The trading partner provides the certificates necessary to encrypt and verify AS2 messages exchanged with them. Arc accepts X.509 public key certificates (files with .cer, .der, or .pem extensions). Typically the trading partner provides a single certificate, which should be configured in the **Encryption Certificate** field. If the trading partner provides multiple certificates, they should clarify the purpose of each certificate. If the partner provides a full certificate chain (as acquired from a commercial certificate authority), only the leaf certificate (the last certificate in the chain) needs to be configured. Rarely, a separate public key certificate might be required to verify the partner's digital signatures. In this case, set the signature verification certificate on the **Advanced tab** > **Advanced Settings** > **Partner Signing Certificate**. ## Send and Receive Files Once the AS2 profile and partner-specific AS2 connectors have been configured, files can be securely sent and received. ### Send Files In an AS2 connector, the **Transactions** tab displays the files to be sent to the target trading partner. If **Send Automation** is enabled on the [Automation](#automation-tab) tab, files that reach the **Transactions** tab of the connector are automatically packaged and sent. Access the log files for all transmissions by expanding the row associated with the transmitted file. The **Create Test Files** button lets you generate a simple series of test files to send to the trading partner. ### Resend and Retry An AS2 Resend is triggered when the trading partner is expected to return an asynchronous MDN, but fails to do so within the **Resend Interval** duration (60 minutes by default). The application then attempts to resend the transmission. The application continues resending the message until an MDN is received or the **Max Attempts (async)** is exhausted. A Retry is triggered when the HTTP response from the trading partner indicates that the server has not received the transmission (the response is not a positive 200 OK status). This can indicate a networking or connectivity issue, which is often transient. The application retries the transmission every **Retry Interval** minutes until the transmission is received or the **Max Attempts** is exhausted. ### Receive Files In an AS2 connector, the **Transactions** tab displays the files that have been received by the application and routed to the connector (based on the AS2 identifiers present in the incoming AS2 message). Expand each file row to display a list of available logs for the transmission. These files are available on the connector **Transactions** tab. If the connector is connected to other connectors in the flow, files are automatically moved from the **Transactions** tab of the AS2 connector to the **Transactions** tab of the next connector in the flow. The AS2 protocol does not allow for actively pulling files from trading partners: the AS2 connector can only passively wait for a trading partner to send a file. ### Troubleshooting Receiving Files Issues that occur when receiving AS2 messages can be harder to track down than issues that occur when sending files. When an error occurs while *sending* a file, the error is immediately reported on the **Transactions** tab for the AS2 connector and on the [Activity](../getting-started/administration/activity) page. When *receiving* files, errors and other debug information can appear in multiple places. After Arc receives an AS2 message, it attempts to route that message to a specific AS2 connector, based on the **AS2 Identifier** configured in the connector (and the AS2 identifiers present in the incoming message). There are three possible locations to check for logging information, depending on whether this routing operation succeeds: * The **Transactions** tab for the AS2 connector (configured for this trading partner) has error logs if Arc successfully routed the message. * The [Application logs](../getting-started/administration/activity#application-logs) have error logs if Arc could not successfully route the message. * If logs are not present in the AS2 connector *or* the application tab, the AS2 message never reached Arc in the first place. In the last case, where there are no logs at all, the issue is likely related to network configuration. This often occurs as a result of firewall interference, so it is important to confirm that the trading partner and the system hosting Arc have opened the appropriate ports on the firewall and whitelisted IP addresses if necessary. There might also be a simple misconfiguration on the trading partner's side, such as sending to the incorrect endpoint (the partner should send to the **Receiving URL** value found in the **AS2 Profile** tab of the [Profiles](../getting-started/administration/profiles) page). ## What Happens in an AS2 Exchange For all of its complexity in terms of its applications, AS2 boils down to two basic parts: A document is sent from an AS2 sender to an AS2 receiver via HTTP, a very flexible client-server protocol that serves as the backbone of the web. The receiver acknowledges the transfer by providing the sender with a receipt. The following illustration demonstrates the steps in further detail. ![AS2 Diagram](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/AS2-Diagram.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=29e68d4c1af6721697f60c8b90b46a76) ### Step 1: EDI Document Preparation Any type of document can be sent in an AS2 exchange, from a text file to a PDF. Typically, however, most trading partners implement a standard for specific document types. The most common document types are electronic data exchange (EDI) X12 documents (.x12 or .edi files), Electronic Data Interchange for Administration, Commerce and Transport (EDIFACT) documents (.edifact files), or XML files (.xml files). The preparation of the document takes place before the AS2 communication begins. EDI is a general term for the transfer of documents between trading partners and the standards that are adhered to. It is also represented as EDI over the Internet (EDIINT). ### Step 2: AS2 Packaging The AS2 document is prepared for sending. This consists of three types of document transformation: * If the document consists of data that is compressible (that is, not binary data), the document can be *compressed* using the zlib compression algorithm to reduce the size of the transported data. * The data is typically *signed* using the private key of the sender to ensure the sender's identity as the creator of the document (usually with the SHA-1 signing algorithm). * Finally, the data can be *encrypted* using the receiver's public key so only the trading partner can read the data (usually using the 3DES encryption algorithm). This step can be skipped if the data is going to be delivered by a secure transport mechanism, such as HTTPS. S/MIME is the set of standards used for encryption and signing of a message/document. It not only governs the functions of signing and encryption, but also provides standards for the formatting of the final message so that a compliant reader can easily identify the structure of the message. ### Step 3: HTTP/S Delivery The prepared document is delivered to the trading partner over the internet to the trading partner's web server over the HTTP or HTTPS protocol. ### Step 4: AS2 Unpackaging The receiver of the prepared document unpackages it to retrieve the EDI document. If the data is encrypted, the prepared document is *decrypted* using the receiver's private key. If the data is signed, the signature on the document is *verified* using the sender's public key to ensure the identity of the sender. If the document is compressed, the prepared document is *decompressed* to produce the original EDI document. ### Step 5: EDI Processing The AS2 receiver passes the unpackaged EDI document to a back-end process that handles the data to perform any additional business logic. The EDI document is parsed and the receiver might initiate a new AS2 transaction where the roles of sender and receiver are reversed. In particular with EDI-X12 documents, a 997 functional acknowledgment is often sent to the original sender to signify that the original EDI document was processed in the back-end business logic. ### Step 6: MDN Reply The receiver sends an MDN to the sender, usually *signed* with the receiver's private key. The MDN is a receipt that is returned in an AS2 exchange and used to report to the sender what was received and whether or not it was received successfully. The MDN contains information about whether the document was successfully unpackaged, as well as a message digest that is calculated over the received payload. The MDN is then returned to the sender in one of two ways, depending on how the sender asked for the MDN to be delivered. In a *synchronous* transaction, the receiver returns the MDN in the HTTP reply from the receiver's web server. In an *asynchronous* transaction, the HTTP reply contains a simple acknowledgment (200 OK), and the MDN is returned over a separate connection (this is usually the case if the un-packaging of the AS2 transmission is expected to take a while). ### Step 7: MDN Processing When the sender receives the MDN from the receiver, the MDN signature is *verified* if the MDN was signed. The status of the MDN is checked to see if the receiver was successful in processing the transaction, or if it encountered an error that was reported in the MDN. Finally, the message digest reported in the MDN is matched against a message digest that was calculated over the EDI data that was sent. With a signed MDN, the sender can verify that the receiver of the message received the entire contents of the EDI document as it was intended to be delivered. When **Log MDNs** is enabled on the **Advanced** tab, Arc stores the MDN in the `.mdn` log for the transaction, where it serves as proof of receipt. ## Certificates The AS2 connector makes use of both private and public key certificates. ### Private Key Certificates The AS2 connector allows you to specify certificates in PKCS#12 format (a .pfx file or a .p12 file). Private key certificates are used to perform two operations that only the holder of the private key should be able to perform: * *Signing* data to provide proof of your identity * *Decrypting* data that was intended for you ### Public Key Certificates Public key certificates are given to you by your trading partner. The AS2 connector allows you to specify public key certificates in X.509 format (a .cer or .der file). Public key certificates are used to work in reverse of the operations that require a private key. These operations include: * *Verifying* signatures created by your trading partner * *Encrypting* data so that only your trading partner can read it ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Common Errors Following is a list of common errors, their causes, and the recommended solution. Contact [arcsupport@cdata.com](mailto:arcsupport@cdata.com) for more information. ### Error: The receipt signature could not be verified: Message digest mismatch in signature **Cause** MDN receipt signatures contain a message digest that ensures the content of the message has not been altered in transit. A digest mismatch could suggest that the MDN was altered before being received, or was not generated properly on the partner's end. Sometimes this error can be caused by anti-virus or file security software improperly stripping part of the MDN response. There is a known issue with ESET's Application Protocol Filtering feature where folded MDN headers can be invalidated due to whitespace removal. **Resolution** Look at the MDN returned by the partner for obvious issues. Download the .mdn file for the transaction where this error is thrown and send it to [arcsupport@cdata.com](mailto:arcsupport@cdata.com) along with a description of the issue, or investigate independently. It might also be helpful to reach out to the trading partner to confirm what AS2 solution they use. Arc is interoperable with any Drummond-certified AS2 solution. ### Error: The receipt signature could not be verified: The certificate specified does not match the signature **Cause** MDN receipts are signed with a private key, and this signature is validated with the corresponding public key. This error suggests that the public key configured to verify signatures from this partner is incorrect. **Resolution** Often, the same public key certificate used for encryption is also used to verify signatures. In this case, check the certificate set under **Settings** > **Encryption Certificate** in the AS2 connector to make sure it is correctly configured for this trading partner. Sometimes trading partners use a separate certificate for signatures. In this case, set the **Advanced** > **Partner Signing Certificate** setting to the public key certificate that matches the partner's signing key. ### Error: The receipt signature could not be verified: Message digest was encrypted with unknown algorithm **Cause** This is a known issue in older versions of the application that only supported SMIME encryption v3.1. If an MDN contains a message digest encrypted with SMIME 3.2, this error is thrown. **Resolution** All current versions of Arc, including the final published build of v2016, support SMIME 3.2. Older versions require an upgrade to resolve this error. ### Error: MDN Error Authentication failed *OR* Signature Authentication failed: Could not authenticate signer's identity **Cause** This error is included as part of the MDN response, indicating that the trading partner could not verify the signature in the AS2 request. This suggests a general certificate mismatch between the trading parties. **Resolution** Check that the private certificate configured in the Arc AS2 profile is correct, and that the trading partner has configured the matching public key certificate on their end. ### Error: MDN Error: Unexpected processing error **Cause** This error is thrown by the trading partner's system and is returned as part of the MDN to indicate a failure to process the AS2 request. This is a general error, thrown when the problem is not related to signing, encryption, or compression. **Resolution** Since this error does not include specific debugging information, contact the trading partner for further information about what caused the failure. ### Error: System error: Connection refused **Cause** This network error indicates a general connectivity problem. It is thrown when a connection attempt is made to a server that is not actively listening. This might indicate that the server has gone down, or that the connection attempt is being made to the wrong URL. **Resolution** Check that the target URL is correct. If the error persists, contact the server administrators for the target system to see if the server has gone down, or if they have more information about the issue. ### Error: Connection failed A connection attempt failed because the connected party did not properly respond after a period of time *OR* Connection timed out **Cause** This network error indicates a general connectivity problem. It is thrown when the connection attempt to the server goes unanswered for a period of time (typically 60 seconds). This is often the result of firewall interference with the server's response, but it might also indicate incorrect connection parameters. **Resolution** Check that firewalls are not preventing the server's response from being returned. If the server is also behind a firewall, check that the IP from which the AS2 messages are being sent is whitelisted in the server's firewall. If firewalls have been ruled out, check that the target URL is correct. ### Error: Synchronous MDN expected but not received **Cause** This is a general error indicating that the response to the AS2 request was not an MDN. This might indicate that the AS2 request was not sent to an AS2 server as expected. This error can also be thrown if an MDN was returned, but it was corrupted such that Arc cannot appropriately recognize it as an MDN. **Resolution** Check that the target URL is correct. Then check to see if a firewall might be stripping contents from the MDN, preventing Arc from parsing it correctly. If the issue with the MDN is unclear, find the .mdn file associated with the failed transaction and provide it to [arcsupport@cdata.com](mailto:arcsupport@cdata.com) along with a description of the issue. ### Error: Unsigned MDN received, but signed MDN requested **Cause** This error is thrown when the response from the trading partner is expected to be a signed MDN receipt, but is instead some other content. This could be an unsigned MDN receipt, or a response that is not an MDN at all. **Resolution** Check the contents of the MDN log file from the failed transaction. This file contains the server reply, whether it is an unsigned MDN or some non-MDN response. The contents of the MDN log file indicate the next steps: if the server response is not an MDN at all, it might contain an error or indicate that the endpoint where the AS2 request was sent is not an AS2 receiving endpoint. If the MDN log file contains an unsigned MDN, this indicates a configuration issue in the trading partner's system. ### Error: Unable to find valid certification path to requested target **Cause** This error is thrown in the Java edition by the underlying Java security provider. It indicates that the SSL server certificate presented by the target web server is not trusted by the system. **Resolution** The TLS server certificate trust can be overridden in the AS2 connector by setting the **Trading Partner Certificates** > **TLS Server Certificate** setting to the server's public key certificate or to `Any Certificate`. You can acquire the server certificate by connecting to the web server through most web browsers. ### Error: Key does not exist **Cause** This is a known issue in the Windows CryptoAPI when multiple threads are attempting to access the same private key. By default, Arc uses the Windows CryptoAPI to perform security operations like loading private certificates. **Resolution** Arc comes with an internal implementation of crypto operations, and this managed implementation can be enabled to work around the Windows CryptoAPI limitation. Navigate to the `data` folder in the Arc installation directory, find the folder for the relevant AS2 connector, and open `port.cfg` in a text editor. Make sure the following line is present in order to enable the managed security implementation while loading certificates: ``` CertUseManagedSecurityAPI = true ``` ### Error: Input string was not in a correct format **Cause** This indicates that string validation failed for some aspect of the AS2 connector configuration. Typically the **Receiving URL** setting or some string processing in the Events scripts are the source of the error. **Resolution** Check that the target URL in **Receiving URL** begins with a valid HTTP prefix (`http://` for plain text connections or `https://` for SSL connections) and does not contain any unintended whitespace. If scripts are present in the Events of the connector (for example, `BeforeSend` or `AfterReceive`), check that invalid string processing is not being performed. Contact [arcsupport@cdata.com](mailto:arcsupport@cdata.com) to confirm whether the scripts are causing the issue. ### Error: 500 Internal Server Error **Cause** This HTTP error indicates a general server-side failure. The AS2 message was successfully received, but an error occurred when processing the message and no further debugging information is available. **Resolution** The only client-side setting that could be related to this issue is the target URL. If this is correct, consult the server-side logs to learn more about what caused the processing failure. Consult with the trading partner to see if these server logs are available. ### Error: 404 Not Found **Cause** This HTTP error indicates that no resource could be found at the target URL. This usually indicates that the first half of the URL is correct (host, port) but that the second half of the URL (the resource path) is not recognized. **Resolution** Check that the resource path in the target URL is correct. Confirm with the trading partner what the expected resource path is. ### Error: 401 Unauthorized **Cause** This HTTP error indicates that authorization is required to reach the specified URL. This can be TLS client authentication or HTTP authentication. **Resolution** If TLS client authentication is required, set the appropriate certificate in the **Advanced** > **TLS Client Authentication** section of the connector configuration. If HTTP authentication is required, set the appropriate credentials in **Advanced** > **HTTP Authentication**. To see if HTTP Authentication is required, test a connection to the target URL with a web browser and notice any prompts for user/password credentials. ### Error: Incoming request did not match a configured trading partner profile **Cause** When an AS2 message is received by Arc, the application attempts to route the message to a specific AS2 connector based on the configured AS2 identifiers. This error indicates that no connector could be found where the AS2 identifiers for sender and receiver match the values present in the AS2 message. **Resolution** Check the AS2 identifiers configured on both the AS2 [Profiles](../getting-started/administration/profiles) page and the AS2 connector configured for the specific trading partner affected by this error. ### Error: Error during handshake: The token supplied to the function is invalid **Cause** This is one of several SSL errors returned by the WinSock library (Windows Sockets). This error often occurs when an HTTP connection is attempted with a server that expects HTTPS connections. **Resolution** Confirm with the trading partner whether the AS2 connection should be over HTTP or HTTPS, and check that the target URL has the appropriate prefix and port. ### Error: Error during handshake: the buffer supplied to a function was too small **Cause** This is one of several SSL errors returned by the system WinSock library, and is a known bug with the Windows CryptoAPI on certain server OSes. The issue occurs when using TLS 1.2 and two specific cipher suites: * `TLS_DHE_RSA_WITH_AES_128_GCM_SHA256` * `TLS_DHE_RSA_WITH_AES_256_GCM_SHA384` **Resolution** Arc comes with an internal implementation of crypto operations, and this managed implementation can be enabled to work around the Windows CryptoAPI limitation. Navigate to the `data` folder in the Arc installation directory and open the `profile.cfg` file in a text editor. Make sure the following line is present under the `[Application]` section: ``` UseManagedSecurityAPI = true ``` As an alternative to using the managed security provider, use a third-party system crypto tool to disable the two affected ciphers at the OS level. ### Error: Error during handshake: the function requested is not supported **Cause** This is one of several SSL errors returned by the system WinSock library, and indicates a mismatch in the supported TLS/SSL versions between client and server. **Resolution** TLS versions can be enabled in the AS2 connector under **Advanced** > **Other Settings** > **TLS Enabled Protocols**. Confirm with the trading partner which TLS/SSL versions are supported and configure the connector appropriately. # AS3 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/as3 Configuration and usage guide for the CData Arc AS3 connector, which supports sending and receiving files using the AS3 protocol over FTP with AS2-style security. AS3 connectors support sending and receiving files using the Applicability Statement 3 (AS3) protocol. ## Key Capabilities * Secure file transfer combining FTP protocol with [AS2](./as2)-style security * Digital encryption and signature support for enhanced security * MDN receipt generation and processing for reliable delivery confirmation * Support for both upload and download operations with filtering * TLS/SSL support with client authentication capabilities ## Overview Each AS3 connector is configured to exchange files with a single AS3 trading partner. AS3 connectors use the FTP protocol to transfer files, and add security and verification through digital encryption and signatures. Configuring an AS3 trading relationship requires exchanging AS3-specific profile details, such as AS3 identifiers and digital certificates, and the connection details for the target FTP server. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Trading Partner Info *Settings for identifying and connecting to a specific AS3 trading partner.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **AS3 Identifier** | The AS3 identifier specific to the target trading partner. | #### Local Profile *Settings that identify the local AS3 profile.* | Setting | Description | | - | - | | **Local AS3 Identifier** | Your AS3 identifier. | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. | | **Certificate Password** | The password required to access the local private certificate. | #### Connection Info *Settings related to connection parameters for the target FTP server.* | Setting | Description | | - | - | | **Host** | The hostname or IP address of the FTP server. | | **Port** | The port on which to connect to the FTP server. | | **User** | The username credential for logging in to the FTP server. | | **Password** | The password credential for logging in to the FTP server. | | **TLS Type** | How to negotiate TLS/SSL when connecting to the server. Choose **Explicit** to establish a plain text connection where TLS/SSL is then started with an explicit command. Choose **Implicit** to immediately negotiate TLS/SSL without establishing a plain text connection. | | **Send Message Security** | Whether to sign and/or encrypt outgoing AS3 messages. CData strongly recommends you use signatures and encryption. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming AS3 messages. An error is thrown if a received message does not have a required security parameter. | | **Compression** | Whether to compress the payload of outgoing messages. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | #### MDN Receipts *Settings related to requesting MDNs when sending AS3 messages.* | Setting | Description | | - | - | | **Request MDN Receipt** | Whether an MDN receipt should be returned in response to outgoing AS3 messages. CData strongly recommends that you request MDN receipts. | | **MDN Path** | The folder path on the server where the partner uploads MDN receipts. | | **Security** | Whether the MDN receipt should include a signature block verifying the message integrity and identity of the recipient. Again, CData strongly recommends you use this option. | #### Upload *Settings related to how outgoing AS3 messages are uploaded.* | Setting | Description | | - | - | | **Remote Path** | The folder path on the remote FTP server where outgoing messages should be uploaded. | #### Download *Settings related to how incoming AS3 messages are downloaded.* | Setting | Description | | - | - | | **Remote Path** | The folder path on the remote FTP server from which incoming messages should be downloaded. | | **File Mask** | A glob pattern that determines which files within the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete files** | Whether files should be deleted from the FTP server after they are successfully downloaded. | #### Trading Partner Certificates *Settings related to the public certificates provided by the trading partner.* | Setting | Description | | - | - | | **Encryption Certificate** | The public key certificate used for AS3 encryption when sending messages. This certificate must be paired with the trading partner's private certificate, and the trading partner should provide a public key certificate when sharing AS3 configuration details. | | **Verification Certificate** | The public certificate used to verify AS3 signatures when receiving messages. This field is often unnecessary; most AS3 parties use the same private certificate for both signing and decrypting. If this field is not specified, the application uses the **Encryption Certificate** to verify signatures. | | **TLS Server Certificate** | The public certificate used to verify the identity of an SSL/TLS server. This is only necessary if the FTP server requires FTPS (FTP over TLS/SSL). If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or it can be set to `Any Certificate` to unconditionally trust the target server's identity. | ### Advanced Tab #### Local Profile *Settings related to client authentication when two-way SSL authentication is required.* | Setting | Description | | - | - | | **Use Profile** | Whether to use the **Private Certificate** configured on the Profiles page as the TLS certificate for client authentication. | | **Private Certificate** | The private certificate presented during TLS client authentication. Only applicable if you are not using the same private certificate from the Profiles page. | | **Certificate Password** | The password required to access the TLS client certificate. | #### Server Commands These options allow you to execute custom commands on the target server at various times during connector execution. > [!NOTE] > > While the commands described below are commonly accepted, be sure to check with the target server administrator to ensure that the server can accept a command before trying to use it.
SettingDescription
**After Connect**Execute this command directly after connecting.
**After Get**Execute this command after downloading each file.
**After Put**Execute this command after uploading each file.
**Before Get**Execute this command before downloading each file.
**Before Put**Execute this command before uploading each file.
Following are the commands you can submit to the target server when you include them in the fields described above: • `onerror`: Pass the `fail` argument to instruct the connector to treat errors that come from the custom scripts as errors for the overall transaction. Without `fail` the connector logs any exceptions and continues processing. • `cd`: Change directory. • `mkdir`: Create directory. • `rn` or `move`: Rename or move file. • `del` or `rm`: Delete or remove file. • `rmdir`: Remove directory. CData Arc supports the following macros with these commands. They all use the following syntax: `%Macro%`. Some macros, such as `%Date%` and `%SourceFilename%`, do not require an argument, but others do. The macros that take an argument use the following syntax: `%Macro:argument%`.
MacroDescription
MsgHeader:*headername*Evaluates to the header value of the message (for example: %MsgHeader:headername%).
SourceFilenameEvaluates to the filename (extension included) of the file currently being processed.
SourceFilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed.
DateEvaluates to the date the file was last modified in a yyyy-MM-dd format (for example, 2024-02-13).
DateTimeEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss format (for example, 2024-02-13T08:14:22).
TimeEvaluates to the time the file was last modified in a HH:mm:ss format (for example, 08:14:22).
DateTime.tzEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss zzz format (for example, 2024-02-13T08:14:22 PST).
Time.tzEvaluates to the time the file was last modified in a HH:mm:ss zzz format (for example, 08:14:22 PST).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
**Example**: After downloading, you want to move each file from the download folder to a History subfolder. Use the **After Get** field to supply the `move` command with the `%SourceFilename%` macro. ![Server command example](/26.4/images/server_command_example.png) #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Active Mode** | Whether to enable Active or Passive mode on the remote FTP server. Passive mode is less likely to result in firewall interference with the connection. | | **Clear Command Channel** | Check this to send FTP commands using a cleartext channel instead of an encrypted channel. | | **Clear Data Channel** | Check this to have the application use a clear data channel when communicating with the FTP server. | | **Encryption Algorithm** | The algorithm to use when encrypting outgoing AS3 messages. | | **EPSV** | Check this to use extended passive mode. This might be necessary if you need to use a protocol other than IPv4. | | **Excluded Extensions** | A comma-delimited list of file extensions that should not be processed by the connector. | | **FSwitch** | Used in conjunction with **Simple Dir List**. Check this to restrict the directory listing to files only. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Prot for Implicit TLS** | Tells the server that the data channel is protected for implicit TLS connections. Select the TLS type on the **Settings** tab. | | **Remote Host Address for Passive** | Only applicable when **Active Mode** is not set. Leave blank to have the application parse the remote host to send replies from the previous server response. When this is checked, the application uses the value of the RemoteHost setting instead. | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. The same algorithm is requested for the corresponding MDN receipts. | | **Simple Dir List** | Check this when you need a simple directory listing command for servers that send back long responses that cannot be parsed. After you set this, if the connector is still unable to parse the directory listing, add the **FSwitch** setting to restrict the directory listing to files only. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Upload** | Whether files arriving at the connector should automatically be uploaded. | | **Retry Interval** | The number of minutes before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement, and validation of the receipt (when requested synchronously). If you set this to 0, the connect retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous MDN receipt is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path(s) for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection To connect to the remote FTP server, specify the **Host**, **Port**, **User**, **Password**, and **TLS Type** required by the server. Use the Test Connection button to verify that the connection parameters are correct. Once connected, set the upload and download **Remote Path** so that files are placed in and read from the appropriate location to communicate with the trading partner. ## Automatic Cache Cleanup Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. The same cache cleanup process works with [MDN receipts](#mdn-receipts). ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# AS4 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/as4 Configuration and usage guide for the CData Arc AS4 connector, which supports sending and receiving messages using the AS4 protocol. The AS4 connector supports sending and receiving messages using the Applicability Statement 4 (AS4) protocol. ## Key Capabilities * Modern SOAP-based B2B messaging with multiple profile support (ENTSOG, e-SENS, DBEW, and eDelivery) * Advanced security with token authentication and certificate management * Support for both push and pull messaging patterns * Business agreement handling with service and action definitions * Comprehensive message properties and custom headers support * Certified by the [Drummond Group](https://www.drummondgroup.com/certified-products/b2b-interoperability/) ## Overview An AS4 connection is configured in two places. Configure the AS4 [Profiles](../getting-started/administration/profiles) page with a local AS4 party identifier and a private certificate. Then configure individual AS4 connectors with connection settings specific to a single trading partner. When an input file is processed by an AS4 connector, it is packaged and sent to the specified trading partner. When Arc receives a file over AS4, it attempts to route the file to a specific AS4 connector. The application uses the AS4 party identifiers in the AS4 message to determine which AS4 connector should receive the file. When a file is routed to an AS4 connector, that file is placed in the connector's Transactions tab, or is passed along to the next connected connector in the flow. The AS4 connector supports shared connections. This means that connections and configurations made in one connector also appear on the **Settings > Connections** tab so they can be reused across other AS4 connectors. ## Profile Configuration The AS4 Profile must be configured before connections can be established with individual AS4 connectors. Click **Profiles** on the navbar, then click the **AS4** tab. ### AS4 Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **Party Identifier** | Your AS4 identifier. Messages sent by Arc must be addressed to this identifier, and outgoing messages use this value to identify the originator of the message. Identifiers are case-sensitive. | | **Party Identifier Type** | The optional type declaration of the **Party identifier**. If specified, supply the domain to which the identifier belongs. | #### Personal Certificate *Settings related to the private decryption certificate and an optional signing certificate.* | Setting | Description | | - | - | | **Private Certificate** | The private certificate used to decrypt incoming messages and, unless a separate **Signing Certificate** is set, to sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in an AS4 transaction: a corresponding public key is also generated with the same filename and a `.cer` extension. Accepts PKCS#12 certificates (.pfx or .p12). | | **Private Certificate Password** | The password required to access the **Private Certificate**. | | **Signing Certificate** | An optional private certificate used to sign outgoing messages. If set, it is used for signing instead of the **Private Certificate**. Set this when your trading partner requires separate signing and decryption certificates, such as with the eDelivery V2 or ENTSOG V4 profiles. | | **Signing Certificate Password** | The password required to access the **Signing Certificate**. | #### Rollover Certificate | Setting | Description | | - | - | | **Rollover Private Certificate** | A secondary private certificate that can be used to decrypt incoming messages if decryption with the Private Certificate fails. Only use this setting when transitioning certificates when an overlap period is necessary to accept incoming messages encrypted with either the old or the new certificate. The rollover certificate is never used to sign outgoing messages, which can cause signature verification issues when sending to trading partners that are still using the old certificate. | | **Rollover Private Certificate Password** | The password required to access the **Rollover Private Certificate**. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Rotate Interval** The number of days to wait before creating a new log file.
**Log Delete Interval** The number of days to wait before deleting old log files.
#### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Receiving URL** | The URL at which Arc listens for incoming AS4 messages. Share this URL with all your trading partners. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration Once you configure the global AS4 profile settings, create and configure individual AS4 connectors for each trading partner on the [Flows](docs/flows/flows) page. ### Settings Tab #### Configuration | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Connection** | A list of available AS4 connections. Select an existing connection or click **Create** to add a new one. See [Create a Connection](#create-a-connection) for descriptions of the options on the **Add Connection** page. | #### Connection Info *Settings related to connection parameters for the specified trading partner.* | Setting | Description | | - | - | | **Send Message Security** | Whether to sign and/or encrypt outgoing AS4 messages. CData strongly recommends you use signatures and encryption. | | **Receive Message Security** | Whether to require that signatures and/or encryption are present for incoming AS4 messages. An error is thrown if a received message does not have a required security parameter. | | **Compression** | Whether to compress the payload of outgoing messages. | | **Message Partition Channel** | The channel over which the AS4 interchange should occur. If you do not specify a channel, the default Message Partition Channel (MPC) is implicitly assigned. MPCs are important when sending or receiving pull requests, because only messages queued in the assigned MPC are returned in response to the pull request. | #### Receipt *Settings related to requesting receipts when sending AS4 messages.* | Setting | Description | | - | - | | **Enable Receipt** | Whether a receipt should be returned in response to incoming AS4 messages, and expected in response to outgoing AS4 messages. | | **Delivery** | Whether the receipt should be returned as a direct response to the outgoing AS4 message (Synchronous) or returned later as part of a separate connection (Asynchronous). Synchronous receipts are recommended unless the size of the AS4 message is very large (50MB is a common threshold), in which case processing the message and delivering a synchronous receipt might strain the connection timeout duration. | #### Business Agreement Details *Settings related to the shared details of the AS4 business agreement. These values should be confirmed with the trading partner prior to configuration.* | Setting | Description | | - | - | | **Service** | The business process accepting the message. | | **Service Action** | The operation or activity being executed in the business process. | | **Service Type** | The optional qualifier providing context to the **Service**. | | **From Role** | The role of the party from which the message originates. These roles can either be reversed or remain the same when sending versus receiving AS4 requests, based on your partner agreement. This value is used in the `PartyInfo` element of the resulting AS4 SOAP envelope. | | **To Role** | The role of the party to which the message is delivered. These roles can either be reversed or remain the same when sending versus receiving AS4 requests, based on your partner agreement. This value is used in the `PartyInfo` element of the resulting AS4 SOAP envelope. | #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message Properties You can provide a set of custom message properties, specified in name-value pairs, to include as part of the outgoing message. Setting the name-value pairs in the UI is the equivalent of doing `` on the message before it reaches the AS4 connector. You can override message properties dynamically based on message content, or specify a type other than `string`. To accomplish that, use a script similar to the following to add headers: ```xml theme={null} ``` The names specified in `AS4-MessagePropertyType` must match the names used in `AS4-MessageProperty`. Multiple properties are separated by semicolons. For example, the following expands the script above to include an `integer` property: ```xml theme={null} ``` #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Attach XML Files** | Whether XML payloads should be packaged as a MIME attachment to the SOAP message, or included directly in the SOAP body. | | **Duplicate Message Action** | How the connector should behave when it receives an AS4 message with a Message Id that the connector has seen before. The connector detects duplicate messages by *remembering* Message Ids that have been received within the number of minutes specified in **Duplicate Interval**. Options: **Continue** — Processes the duplicate file like any other file. **Ignore** — Ignores the inbound file and does not write it to the Receive folder. **Failure** — Raises an error and does not process the file. | | **Duplicate Interval** | The number of minutes that a message with the same Message Id is considered a duplicate. If you set this to 0, the Message Ids are stored until the server is restarted. | | **Pull URL** | The URL endpoint to which pull requests should be sent. Only required if it is different from the **Partner URL**. | | **Receipt URL** | The URL endpoint to which receipts should be sent. Only required if it is different from the **Partner URL**. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as AS4 messages. | | **Retry Interval (minutes)** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connect retries the file indefinitely. | | **Resend Interval (minutes)** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous receipt is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | | **Receive** | Whether the connector should automatically make pull requests to receive messages from the trading partner. | | **Interval** | The interval between automatic pull requests. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Create a Connection When you create a new connection, begin by supplying a **Name**, then complete the following settings. *Settings for identifying and connecting to a specific AS4 trading partner.* | Setting | Description | | - | - | | **Party Identifier** | The AS4 party identifier specific to the target trading partner. This value is used in the header of outgoing requests to indicate the intended recipient, and it is used to route incoming AS4 messages to the appropriate AS4 connector. Identifiers are case-sensitive. | | **Party Identifier Type** | The optional type declaration of the **Party identifier**. If specified, supply the domain to which the identifier belongs. | | **URL** | The trading partner's public endpoint where outgoing AS4 messages are sent. | | **Agreement** | The shared AS4 Agreement that governs the exchange. This is required in ENTSOG profiles. | | **Connection Timeout (seconds)** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Profile** | The AS4 profile to use in the AS4 interchange. Profiles help determine shared configuration settings to ensure compatibility between partners. The available profiles are: `Standard`, `ENTSOG`, `e-SENS`, `BDEW`, `ENTSOG V4`, and `eDelivery V2`. If you use BDEW, some additional configuration is required. See [Configure a BDEW Profile](#configure-a-bdew-profile) for details. Several advanced settings, including **Encryption Algorithm**, **Signature Algorithm**, **Security Token Format**, and **SSL Enabled Protocols** allow you to select a value of `Auto`. When you use `Auto`, the connector automatically chooses the appropriate value for each setting based on the profile setting. You can still manually override these values if needed. | | **Encryption Certificate** | The public key certificate used for AS4 encryption when sending messages. This certificate must be paired with the trading partner's private certificate, and the trading partner should provide a public key certificate when sharing AS4 configuration details. | | **Verification Certificate** | The public key certificate used to verify AS4 signatures when receiving messages. This field is often unnecessary; most AS4 parties use the same private certificate for both signing and decrypting. If this value is not specified, the application uses the **Encryption Certificate** to verify signatures. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or it can be set to `Any Certificate` to unconditionally trust the target server's identity. | | **Rollover Verification Certificate** | A secondary certificate that can be used to verify message signatures if verification with the **Verification Certificate** fails. Only use this setting when transitioning certificates when an overlap period is necessary. | ### Advanced Connection Settings Click the **Advanced** tab to set the following connection options. #### Token Authentication *Settings related to using and requiring token authentication.* | Setting | Description | | - | - | | **Token Authentication (Receiving)** | Whether authentication tokens must be included in incoming AS4 messages. If enabled, incoming messages must include username credentials that match the **Username** and **Password** fields set under this option. | | **Token Authentication (Sending)** | Whether authentication tokens should be included in outgoing AS4 messages. If enabled, outgoing messages include username credentials that match the **Username** and **Password** fields set under this option. | | **Password Type** | Whether username tokens should be in plain text or encrypted digest format. | #### Alternate Local Profile *Settings that override the AS4 configuration on the **Profiles** page for this specific AS4 connector. Setting an alternate local profile lets you use different local certificates and identifiers for certain trading partners.* | Setting | Description | | - | - | | **Party Identifier** | Your AS4 identifier. Overrides the **Party Identifier** on the Profiles page. | | **Party Identifier Type** | Your AS4 identifier type (optional). Overrides **Party Identifier Type** on the Profiles page. | | **Private Certificate** | The private certificate used to decrypt incoming messages and, unless a separate **Signing Certificate** is set, to sign outgoing messages. Overrides the **Private Certificate** on the Profiles page. | | **Private Certificate Password** | The password required to access the local private certificate. | | **Signing Certificate** | An optional private certificate used to sign outgoing messages. If set, overrides the **Signing Certificate** on the Profiles page. | | **Signing Certificate Password** | The password required to access the **Signing Certificate**. | | **Certificate Subject** | Specifies the distinguished name (DN) of the local certificate in the certificate store to use when signing or decrypting messages. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Use Profile Settings** | Whether to use the **Private Certificate** configured on the Profiles page as the TLS certificate for client authentication. | | **Private Certificate** | The private certificate presented during TLS client authentication. Only applicable if you are not using the same private certificate from the Profiles page. | | **Private Certificate Password** | The password required to access the TLS client certificate. | | **Certificate Subject** | Specifies the distinguished name (DN) of the local certificate in the certificate store to use for TLS authentication. | #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Agreement Ref P-Mode** | The processing mode attribute of the shared AS4 agreement. If you use this, provide the `PMode.Id` parameter. | | **Receive Agreement Ref P-Mode** | The processing mode attribute of the shared AS4 receive agreement. If you use this, provide the `PMode.Id` parameter. | | **Agreement Ref Type** | The reference type attribute of the shared AS4 agreement. The two parties should decide upon the semantics of this value. | | **Encryption Algorithm** | Which encryption algorithm to use if you require message encryption. | | **Security Token Format** | Specifies the security token format that is included in the message for both encrypting and signing. Options: **Auto** (automatically chooses based on Profile). **X.509** (subject in DN format). **Binary** (Base64 encoded public key). **KeyIdentifier** (X.509 Subject-Key-Identifier format). **BinaryPKIPath** (base64 encoded ordered list of X509 public certificates). | | **Signing Security Token Format** | Specifies the security token format for signing (optional). If set, this value is used for signing instead of the value of **Security Token Format**. Options: **Auto** (automatically chooses based on Profile). **X.509** (subject in DN format). **Binary** (Base64 encoded public key). **KeyIdentifier** (X.509 Subject-Key-Identifier format). **BinaryPKIPath** (base64 encoded ordered list of X509 public certificates). | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. The same algorithm is requested for the corresponding receipts. When set to `Auto`, the connector automatically chooses the appropriate value based on the selected Profile. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Signing Certificate** | An optional private certificate used to sign outgoing messages for this connector. If set, overrides the **Signing Certificate** configured on the AS4 Profiles page. | | **Signing Certificate Password** | The password required to access and use the private key associated with the **Signing Certificate**. | | **RSA Hash Algorithm** | The cryptographic hash algorithm (such as SHA-256) used in RSA signature operations to create message digests. | | **Reference Hash Algorithm** | The cryptographic hash algorithm (such as SHA-256) used in references to ensure the integrity of individual message parts. | | **Normalize Issuer Subject** | Allows you to standardize the format of certificate issuer and subject distinguished names for consistent certificate validation. The default is `false`. | | **Key Encryption Algorithm** | The algorithm used to encrypt the symmetric session key that protects the message payload (such as RSA-OAEP or RSA-v1.5). | | **MGF1 Hash Algorithm** | The hash algorithm used in the Mask Generation Function 1 (MGF1) as part of RSA-OAEP padding during key encryption. | | **OAEP Params** | Optional parameters used in Optimal Asymmetric Encryption Padding (OAEP) to provide additional randomization during RSA encryption. | | **Transform Receipt** | A setting that applies XML canonicalization or other transformations to receipt and acknowledgment messages before signing. The default is `true`. | | **Sign Pull Request** | Whether to digitally sign AS4 pull request messages sent to retrieve messages from a trading partner's server. | | **Enable Reference Token Type** | Whether to include security token reference information in the message header to specify how certificates and keys are identified. | | **Recipient Key** | The public key or certificate of the message recipient used to encrypt the message payload for confidential transmission. | | **Key Derivation Method** | The algorithm used to derive encryption keys from shared secrets or key agreement protocols in AS4 security processing. | ### Configure a BDEW Profile The BDEW (Bundesverband der Energie- und Wasserwirtschaft, or German Association of Energy and Water Industries) AS4 profile is a messaging specification defined by the German energy and water sector. When you configure an AS4 connector to use the BDEW profile, you must add the following headers before you can pass messages to the connector. ```xml theme={null} ``` ## Personal Profile Configuration Your AS4 profile contains key pieces of information that are used to identify you with trading partners. It is configured on the [AS4 tab](#as4-profile-tab) of the Profiles page. To get started, specify the following information: * [Party Identifier](#party-identifier) * [Personal Certificate](#personal-certificate) ### Party Identifier Similar to an email address in an email exchange, this identifies the partner in an AS4 transmission. Set **Party Identifier** to a value that meaningfully identifies you or your organization. The value set here is sent in the headers of AS4 transmissions initiated by Arc. The Party Identifier is one piece of information exchanged with trading partners when sharing profile information. Trading partners should use this value in their AS4 system so that they can process requests coming from that identifier. ### Personal Certificate The private key certificate is used to decrypt incoming messages and, unless a separate **Signing Certificate** is configured, to sign outgoing messages. The AS4 profile uses a private key certificate in PKCS#12 format (a .pfx or .p12 file). * Click **Upload Certificate** in the **Private Certificate** dropdown to select an existing certificate from disk. * Click **Create Certificate** to create a new self-signed certificate. This creates a new private key certificate and a public key certificate (.cer file) with the same name. Some AS4 profiles, including eDelivery V2 and ENTSOG V4, require separate certificates for signing and decryption. In these cases, configure a **Signing Certificate** in addition to the **Private Certificate**. The signing certificate is a separate private key certificate used only to sign outgoing messages; the private certificate continues to handle decryption. > [!NOTE] > > When a signing certificate is configured, the **Verification Certificate** field must be populated with the public key of the partner's signing certificate, not the private certificate. If they do not match, outgoing messages fail with a signature verification error. ### Receiving URL This is generated based on the **Base URL** in **Settings > Advanced** and the default Arc listening endpoint. Share this value with trading partners so that they can successfully send AS4 messages to Arc. ## Partner Setup Trading partners are responsible for providing specific settings required to connect to their AS4 endpoints. Each AS4 connector must be configured for a single trading partner relationship. The following AS4 configuration options are required: * [Party Identifier](#partner-identifier) * [Partner URL](#partner-url) * [Trading Partner Certificates](#trading-partner-certificates) * [Use Receipt](#use-receipt) * [Service and Service Action](#service-and-service-action) ### Party Identifier The trading partner provides their **Party Identifier**, which is the part of their profile that is used to identify their organization in an AS4 transmission. The identifier is used in the header of outgoing requests to indicate the intended recipient. > [!NOTE] > > This value is case sensitive, so when configuring new trading partners, make sure the capitalization is correct. ### Partner URL The **Partner URL** is the public endpoint where the trading partner's AS4 system listens for incoming messages. After receiving a URL from the trading partner, test the URL with a web browser to detect any networking or connectivity issues. In some cases, the trading partner might provide more than one URL if they use separate URLs for pushing AS4 transmissions, pulling AS4 transmissions, and returning asynchronous receipts. Configure additional URLs on the Advanced tab under **Pull URL** and **Receipt URL**. ### Trading Partner Certificates Each AS4 connector must be configured with the public key certificate(s) for the target trading partner. The trading partner provides the certificates necessary to encrypt and verify AS4 messages exchanged with them. Arc accepts X.509 public key certificates (files with .cer, .der, or .pem extensions). Typically the trading partner provides a single certificate, which should be configured in the **Encryption Certificate** field. If the trading partner provides multiple certificates, they must clarify the purpose of each certificate. If the partner provides a full certificate chain (as acquired from a commercial certificate authority), only the leaf certificate needs to be configured (the last certificate in the chain). Sometimes, a separate public key certificate might be required to verify the partner's digital signatures. In this case, set the signature verification certificate on the [Settings tab](#settings-tab) > **Trading Partner Certificates** > **Verification Certificate**. > [!NOTE] > > If you are using a profile that requires separate signing and decryption certificates (such as eDelivery V2 or ENTSOG V4), your trading partner likely uses a separate signing certificate as well. In that case, confirm with the trading partner which certificate to use for signature verification, and set the **Verification Certificate** to the public key of their signing certificate rather than their encryption certificate. If the trading partner requires TLS/SSL connections, they can provide you with the public key certificate for their TLS server. This should be set under **TLS Server Certificate**. Alternatively, you can set this field to Any Certificate to implicitly trust the identity of the target TLS server. ### Use Receipt Requesting a receipt ensures non-repudiation of the AS4 message; in other words, the successful delivery of the message cannot be disputed. For this reason, we strongly recommend enabling the **Use Receipt** field. Be sure to agree on the use of receipts with the trading partner ahead-of-time. By default, receipts are delivered synchronously, meaning that they are returned as a direct response to the AS4 message. Asynchronous receipts are delivered later in a separate connection. Synchronous receipts are recommended unless the size of the AS4 messages is very large, in which case processing and delivering a synchronous receipt might strain the connection timeout duration. ### Service and Service Action Service, Service Action, and other business agreement details should be communicated with the trading partner. These values help define the business logic associated with the AS4 messages exchanged and how each party should interpret the messages. ### Import from SMP If the trading partner has a Service Metadata Publisher (SMP) profile, these settings can be imported by clicking the **Import from SMP** button in Settings tab > **Trading Partner Info**. The resulting prompt requires the SML Domain, Participant Id and Scheme, and Document Id and Scheme from which to obtain the SMP profile. The connector settings are automatically populated with the values read from the SMP profile. ## Send and Receive Files After the AS4 profile and partner-specific AS4 connectors have been configured, files can be securely sent and received. ### Send Files In an AS4 connector, the Transactions tab displays the files that should be sent to the target trading partner. If **Send Automation** is enabled on the Automation tab, files that reach the connector Transactions tab are automatically enveloped and sent. Access the log files for all transmissions by expanding the row associated with the transmitted file. The **Create Test Files** button lets you generate a simple series of test files to send to the trading partner. ### Resend and Retry An AS4 Resend is triggered when the trading partner is expected to return an asynchronous receipt, but fails to do so within the **Resend Interval** duration (60 minutes by default). The application then attempts to resend the transmission. The application continues resending the message until a receipt is received or the **Max Attempts (async)** is exhausted. A Retry is triggered when the HTTP response from the trading partner indicates that the server has not received the transmission (the response is not a positive 200 OK status). This can indicate a networking or connectivity issue, which is often transient. The application retries the transmission every **Retry Interval** minutes until the transmission is received or the **Max Attempts** is exhausted. ### Receive Files In an AS4 connector, the Transactions tab displays the files that have been received by the application and routed to the connector (based on the AS4 identifiers present in the incoming AS4 message). Expand each file row to display a list of available logs for the transmission. These files are available on the connector **Transactions** tab. If the connector is connected to other connectors in the flow, files are automatically moved from the **Transactions** tab of the AS4 connector to the **Transactions** tab of the next connector in the flow. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# AS4 V1 Connector (Deprecated) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/as4v1 Configuration and usage guide for the deprecated CData Arc AS4 V1 connector, which supports sending and receiving messages via the AS4 protocol. The AS4 connector supports sending and receiving messages via the AS4 protocol. ## Overview An AS4 connection is configured in two places. Configure the AS4 [Profiles](../getting-started/administration/profiles) page with a local AS4 party identifier and a private certificate. Then configure individual AS4 connectors with connection settings specific to a single trading partner. When an input file is processed by an AS4 connector, it is packaged and sent to the specified trading partner. When CData Arc receives a file over AS4, it attempts to route the file to a specific AS4 connector. The application uses the AS4 party identifiers in the AS4 message to determine which AS4 connector should receive the file. When a file is routed to an AS4 connector, that file is placed in the connector's Output tab, or is passed along to the next connected connector in the flow. ## Profile Configuration The AS4 Profile must be configured before connections can be established with individual AS4 connectors. Click **Profiles** on the navbar, then click the **AS4** tab. ### AS4 Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **Party Identifier** | Your AS4 identifier. Messages sent by Arc must be addressed to this identifier, and outgoing messages use this value to identify the originator of the message. Identifiers are case-sensitive. | | **Party Identifier Type** | The optional type declaration of the **Party identifier**. If specified, supply the domain to which the identifier belongs. | #### Personal Certificate *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in an AS4 transaction: a corresponding public key is also generated with the same filename and a `.cer` extension. | | **Certificate Password** | The password required to access the **Private Certificate**. | #### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Receiving URL** | The URL at which Arc listens for incoming AS4 messages. Share this URL with all your trading partners. | #### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Rotate Interval** | The number of days to wait before creating a new log file. | | **Log Delete Interval** | The number of days to wait before deleting old log files. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration Once you configure the global AS4 profile settings, create and configure individual AS4 connectors for each trading partner on the [Flows](docs/flows/flows) page. ### Settings Tab #### Configuration | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | #### Trading Partner Info *Settings for identifying and connecting to a specific AS4 trading partner.* | Setting | Description | | - | - | | **Party Identifier** | The AS4 party identifier specific to the target trading partner. This value is included in the outgoing AS4 message to ensure it is delivered to the intended recipient, and it is used to route incoming AS4 messages to the appropriate AS4 connector. Identifiers are case-sensitive. | | **Party Identifier Type** | The optional type declaration of the **Party identifier**. If specified, supply the domain to which the identifier belongs. | | **Partner URL** | The trading partner's public endpoint where outgoing AS4 messages should be sent. | | **Agreement** | The shared AS4 Agreement that governs the exchange. This is required in ENTSOG profiles. | #### Connection Info *Settings related connection parameters for the specified trading partner.* | Setting | Description | | - | - | | **Send Message Security** | Whether to sign and/or encrypt outgoing AS4 messages. CData strongly recommends you use signatures and encryption. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming AS4 messages. An error is thrown if a received message does not have a required security parameter. | | **Compression** | Whether to compress the payload of outgoing messages. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Profile** | The AS4 profile to use in the AS4 interchange. Profiles help determine shared configuration settings to ensure compatibility between partners. The available profiles are: `Standard`, `ENTSOG`, `e-SENS`, `DBEW`, `ENTSOG V4`, and `eDelivery V2`. | | **Message Partition Channel** | The channel over which the AS4 interchange should occur. If you do not specify a channel, the default Message Partition Channel (MPC) is implicitly assigned. MPCs are important when sending or receiving pull requests, because only messages queued in the assigned MPC are returned in response to the pull request. | #### Receipt *Settings related to requesting receipts when sending AS4 messages.* | Setting | Description | | - | - | | **Enable Receipt** | Whether a receipt should be returned in response to incoming AS4 messages, and expected in response to outgoing AS4 messages. | | **Delivery** | Whether the receipt should be returned as a direct response to the outgoing AS4 message (Synchronous) or returned later as part of a separate connection (Asynchronous). Synchronous receipts are recommended unless the size of the AS4 message is very large (50MB is a common threshold), in which case processing the message and delivering a synchronous receipt might strain the connection timeout duration. | #### Trading Partner Certificates *Settings related to the public key certificates provided by the trading partner.* | Setting | Description | | - | - | | **Encryption Certificate** | The public key certificate used for AS4 encryption when sending messages. This certificate must be paired with the trading partner's private certificate, and the trading partner should provide a public key certificate when sharing AS4 configuration details. | | **Verification Certificate** | The public key certificate used to verify AS4 signatures when receiving messages. This field is often unnecessary; most AS4 parties use the same private certificate for both signing and decrypting. If this value is not specified, the application uses the **Encryption Certificate** to verify signatures. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the partner's AS4 system requires HTTPS instead of HTTP. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or it can be set to `Any Certificate` to unconditionally trust the target server's identity. | #### Business Agreement Details *Settings related to the shared details of the AS4 business agreement. These values should be confirmed with the trading partner prior to configuration.* | Setting | Description | | - | - | | **Service** | The business process accepting the message. | | **Service Action** | The operation or activity being executed in the business process. | | **Service Type** | The optional qualifier providing context to the **Service**. | | **From Role** | The role of the party from which the message originates. These roles can either be reversed or remain the same when sending versus receiving AS4 requests, based on your partner agreement. This value is used in the `PartyInfo` element of the resulting AS4 SOAP envelope. | | **To Role** | The role of the party to which the message is delivered. These roles can either be reversed or remain the same when sending versus receiving AS4 requests, based on your partner agreement. This value is used in the `PartyInfo` element of the resulting AS4 SOAP envelope. | ### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as AS4 messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connect retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous receipt is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | | **Receive** | Whether the connector should automatically make pull requests to receive messages from the trading partner. | | **Pull Interval** | The interval between automatic pull requests. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Advanced Tab #### Token Authentication *Settings related to using and requiring token authentication.* | Setting | Description | | - | - | | **Token Authentication (Receiving)** | Whether authentication tokens must be included in incoming AS4 messages. If enabled, incoming messages must include username credentials that match the **Username** and **Password** fields set under this option. | | **Token Authentication (Sending)** | Whether authentication tokens should be included in outgoing AS4 messages. If enabled, outgoing messages include username credentials that match the **Username** and **Password** fields set under this option. | | **Password Type** | Whether username tokens should be in plain text or encrypted digest format. | #### Alternate Local Profile *Settings that override the AS4 configuration on the **Profiles** page for this specific AS4 connector. Setting an alternate local profile lets you use different local certificates and identifiers for certain trading partners.* | Setting | Description | | - | - | | **Party Identifier** | Your AS4 identifier. Overrides the **Party Identifier** on the Profiles page. | | **Party Identifier Type** | Your AS4 identifier type (optional). Overrides **Party Identifier Type** on the Profiles page. | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Overrides the **Private Certificate** on the Profiles page. | | **Certificate Password** | The password required to access the local private certificate. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Use Profile Settings** | Whether to use the **Private Certificate** configured on the Profiles page as the TLS certificate for client authentication. | | **Private Certificate** | The private certificate presented during TLS client authentication. Only applicable if you are not using the same private certificate from the Profiles page. | | **Certificate Password** | The password required to access the TLS client certificate. | #### Message Properties A set of custom message properties, specified in name-value pairs, to include as part of the outgoing message. #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Agreement Ref P-Mode** | The processing mode attribute of the shared AS4 agreement. If you use this, provide the `PMode.Id` parameter. | | **Agreement Ref Type** | The reference type attribute of the shared AS4 agreement. The two parties should decide upon the semantics of this value. | | **Attach XML Files** | Whether XML payloads should be packaged as a MIME attachment to the SOAP message, or included directly in the SOAP body. | | **Duplicate Message Action** | How the connector should behave when it receives an AS4 message with a Message Id that the connector has seen before. Options: **Continue** (processes the duplicate file like any other file), **Ignore** (silently ignores/discards the inbound file), **Failure** (raises an error and does not process the file). | | **Duplication Interval** | The number of minutes that a message with the same Message Id is considered a duplicate. If you set this to 0, the Message Ids are stored until the server is restarted. | | **Encryption Algorithm** | Which encryption algorithm to use if you require message encryption. | | **Pull URL** | The URL endpoint to which pull requests should be sent. Only required if it is different from the **Partner URL**. | | **Receipt URL** | The URL endpoint to which receipts should be sent. Only required if it is different from the **Partner URL**. | | **Security Token Format** | Which format to use for security tokens (certificate references). | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. The same algorithm is requested for the corresponding receipts. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Personal Profile Configuration Your AS4 profile contains key pieces of information that are used to identify you with trading partners. It is configured on the AS4 tab of the Profiles page. To get started, specify the following information: * Party Identifier * Personal Certificate ### Party Identifier Similar to an email address in an email exchange, this identifies the partner in an AS4 transmission. Set **Party Identifier** to a value that meaningfully identifies you or your organization. The value set here is sent in the headers of AS4 transmissions initiated by Arc. The Party Identifier is one piece of information exchanged with trading partners when sharing profile information. Trading partners should use this value in their AS4 system so that they can process requests coming from that identifier. ### Personal Certificate The private key certificate is used to sign outgoing messages and to decrypt incoming messages. The AS4 profile uses a private key certificate in PKCS#12 format (a .pfx or .p12 file). * Click **Upload Certificate** in the **Private Certificate** dropdown to select an existing certificate from disk. * Click **Create Certificate** to create a new self-signed certificate. This creates a new private key certificate and a public key certificate (.cer file) with the same name. ### Receiving URL This is generated based on the **Base URL** in **Settings > Advanced** and the default Arc listening endpoint. Share this value with trading partners so that they can successfully send AS4 messages to Arc. ## Partner Setup Trading partners are responsible for providing specific settings required to connect to their AS4 endpoints. Each AS4 connector must be configured for a single trading partner relationship. The following AS4 configuration options are required: * Party Identifier * Partner URL * Trading Partner Certificates * Use Receipt * Service and Service Action ### Party Identifier The trading partner provides their **Party Identifier**, which is the part of their profile that is used to identify their organization in an AS4 transmission. > [!NOTE] > > This value is case sensitive, so when configuring new trading partners, make sure the capitalization is correct. ### Partner URL The **Partner URL** is the public endpoint where the trading partner's AS4 system listens for incoming messages. After receiving a URL from the trading partner, test the URL with a web browser to detect any networking or connectivity issues. In some cases, the trading partner might provide more than one URL if they use separate URLs for pushing AS4 transmissions, pulling AS4 transmissions, and returning asynchronous receipts. Configure additional URLs on the [Advanced tab](#advanced-tab) under **Pull URL** and **Receipt URL**. ### Trading Partner Certificates Each AS4 connector must be configured with the public key certificate(s) for the target trading partner. The trading partner provides the certificates necessary to encrypt and verify AS4 messages exchanged with them. Arc accepts X.509 public key certificates (files with .cer, .der, or .pem extensions). Typically the trading partner provides a single certificate, which should be configured in the **Encryption Certificate** field. If the trading partner provides multiple certificates, they must clarify the purpose of each certificate. If the partner provides a full certificate chain (as acquired from a commercial certificate authority), only the leaf certificate needs to be configured (the last certificate in the chain). Sometimes, a separate public key certificate might be required to verify the partner's digital signatures. In this case, set the signature verification certificate on the [Settings tab](#settings-tab) > **Trading Partner Certificates** > **Verification Certificate**. If the trading partner requires TLS/SSL connections, they can provide you with the public key certificate for their TLS server. This should be set under **TLS Server Certificate**. Alternatively, you can set this field to Any Certificate to implicitly trust the identity of the target TLS server. ### Use Receipt Requesting a receipt ensures non-repudiation of the AS4 message. We strongly recommend enabling the **Use Receipt** field. Be sure to agree on the use of receipts with the trading partner ahead-of-time. By default, receipts are delivered synchronously, meaning that they are returned as a direct response to the AS4 message. Asynchronous receipts are delivered later in a separate connection. Synchronous receipts are recommended unless the size of the AS4 messages is very large, in which case processing and delivering a synchronous receipt might strain the connection timeout duration. ### Service and Service Action Service, Service Action, and other [business agreement details](#business-agreement-details) should be communicated with the trading partner. These values help define the business logic associated with the AS4 messages exchanged and how each party should interpret the messages. ### Import from SMP If the trading partner has a Service Metadata Publisher (SMP) profile, these settings can be imported by clicking the **Import from SMP** button in [Settings tab](#settings-tab) > **Trading Partner Info**. The resulting prompt requires the SML Domain, Participant Id and Scheme, and Document Id and Scheme from which to obtain the SMP profile. The connector settings are automatically populated with the values read from the SMP profile. ## Send and Receive Files After the AS4 profile and partner-specific AS4 connectors have been configured, files can be securely sent and received. ### Send Files In an AS4 connector, the Input tab displays the files that should be sent to the target trading partner. If **Send Automation** is enabled on the [Automation tab](#automation-tab), files that reach the connector Input tab are automatically enveloped and sent. The **Create Test Files** button lets you generate a simple series of test files to send to the trading partner. ### Resend and Retry An AS4 Resend is triggered when the trading partner is expected to return an asynchronous receipt, but fails to do so within the **Resend Interval** duration (60 minutes by default). The application then attempts to resend the transmission. The application continues resending the message until a receipt is received or the **Max Attempts (async)** is exhausted. A Retry is triggered when the HTTP response from the trading partner indicates that the server has not received the transmission (the response is not a positive 200 OK status). This can indicate a networking or connectivity issue, which is often transient. The application retries the transmission every **Retry Interval** minutes until the transmission is received or the **Max Attempts** is exhausted. ### Receive Files In an AS4 connector, the Output tab displays the files that have been received by the application and routed to the connector (based on the AS4 identifiers present in the incoming AS4 message). Expand each file row to display a list of available logs for the transmission. These files are available on the connector Output tab. If the connector is connected to other connectors in the flow, files are automatically moved from the Output tab of the AS4 connector to the Input tab of the next connector in the flow. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Azure Blob Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/azure-blob Configuration and usage guide for the CData Arc Azure Blob connector, which supports uploading to and downloading from Azure Blob cloud storage. The Azure Blob connector supports uploading to and downloading from the Azure Blob cloud storage service. ## Key Capabilities * Cloud storage integration with both Access Key and SAS Token authentication * Container-based file organization with upload/download capabilities * File filtering and masking for selective operations * Overwrite protection and deletion management for uploaded files * Optional caching to ensure only new or updated files are downloaded * Exposure of the uploaded blob's remote path to the After Send event and as a message header ## Overview Each Azure Blob connector connects to a specified Microsoft Azure storage account. Within the remote storage, you set a single container as the upload target, and a list of containers as the download target. Files that reach the Azure Blob connector in the Arc flow are uploaded to the upload target container, and files found in the download target container are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specific file names and/or file extensions. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization *Settings related to the Azure account used to access the Azure Blob service.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Account** | The Azure Blob account to access. | | **Auth Scheme** | Whether to authenticate using an **Access Key** or a **SAS Token**. After you make your choice, provide the **AccessKey** or **SAS Tokens**. | | **AccessKey** | The access credential (provided by Microsoft) for the specified account. | | **SAS Tokens** | The Shared Access Token (SAS) for the Azure Storage account. | #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Container Name** | Files processed by the connector are uploaded to this container. | | **Overwrite** | Whether to overwrite or skip existing files. | #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Container Name** | The container from which files are downloaded. | | **File Mask** | A glob pattern that determines which files in the **Remote Path** should be downloaded (for example, \*.txt). If you need to specify multiple file masks, use the **Receive Filter** on the Advanced tab instead. | | **Delete files (after received)** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse remote subdirectories** | Whether to download files in subfolders of the target remote path. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous Settings *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Establishing a connection to the Azure Blob service requires Azure account credentials: * **Account** * **AccessKey** Use the **Test Connection** button to verify that the account information provided is valid. ## Uploading and Downloading ### Upload To upload files, set the **Container Name** in the Send portion of the **Settings** page to the container where files should be uploaded. Each Azure Blob connector uploads to a single container. Use the **Overwrite remote files** option to indicate whether files that already exist in the container should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. #### After Send Event After a successful upload, the connector provides the location of the uploaded blob to the [After Send event](../scripting/scripting#event-scripting) as the following input: | Input | Description | | - | - | | **RemotePath** | The remote path of the uploaded blob. | The value is the container name followed by the blob name, for example `/mycontainer/folder/example.xml`. The path always begins with a single forward slash and uses forward slashes as separators. Any leading or trailing slashes in **Container Name** are removed, and macros in **Container Name** are resolved. The path is not URL-encoded, so characters such as spaces and parentheses appear as they do in the file name. `RemotePath` is available only in the After Send event. It does not exist in the Before Send event, because the upload has not happened yet. This script writes the remote path of each uploaded blob to the [Application log](../getting-started/administration/activity#application-logs): ```xml theme={null} ``` #### RemotePath Header After a successful upload, the connector also adds a `RemotePath` header to the output message. The header contains the same value that the After Send event receives, and you can view it under **Other Headers** in the message details. Because the header is part of the message, a downstream connector attached on the [**Success Path**](../flows/designing-a-flow#additional-configuration-options) can read it. To show the Success Path, right-click the Azure Blob connector in the flow and choose **Show Success Path**. The downstream connector can use the header to log or act on where the blob was stored without reconstructing the path from the connector settings. For example, a [Script](script) connector on the Success Path can pass the path to a back-end process that retrieves the file later. > [!NOTE] > > The `RemotePath` input and header are set only after a successful upload. They are not set if the upload fails. ### Download To download files, set the **Container Name** in the Receive portion of the **Settings** page to the container from where files should be downloaded. Use a comma-delimited list to specify multiple download containers. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the Advanced tab. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. After configuration is complete, files are downloaded according to the **Receive** Automation settings, or you can manually click the **Receive** button on the **Transactions** tab. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Box Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/box Configuration and usage guide for the CData Arc Box connector, which supports uploading to and downloading from the Box cloud storage service. The Box connector supports uploading to and downloading from the Box cloud storage service. ## Key Capabilities * OAuth-based cloud storage integration with automatic token refresh * Upload and download from the connected Box storage account with recursive subdirectory support * File filtering, overwrite protection, and recursive subfolder support * Comprehensive proxy support for enterprise network environments * Optional caching to ensure only new or updated files are downloaded ## Overview Each Box connector connects to a specified Box storage account. Within the remote storage, you set a single folder as the upload and download targets. Files that reach the Box connector in the CData Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specific file names and/or file extensions. Account authentication uses OAuth, so you must enter account credentials in an external site portal the first time the connector connects. The external portal generates an OAuth token that the connector uses to connect, then the connector handles token refresh automatically. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
Use the **Connect** button to authenticate the connector with the storage service the first time. You are redirected to a site login portal. After you provide the account credentials, an authentication token is returned to Arc and stored for repeat use. #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | Files processed by the connector are uploaded to this folder. | | **Overwrite** | Whether to overwrite or skip existing files. | #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | **File Mask** | A glob pattern that determines which files within the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse remote subdirectories** | Whether to download files in subfolders of the target remote path. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection To establish a connection with the Box connector, use the **Connect** button on the [Settings tab](#settings-tab) to be directed to a login portal site. Enter your account credentials in this portal; it returns an authentication token to the connector. This process is only required the first time the connector connects to the storage service. ## Uploading and Downloading ### Upload To upload files, set the **Remote Folder** in the [Send](#send) portion of the **Settings** page to the folder where files should be uploaded. Each Box connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the container should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Download To download files, set the **Remote Folder** in the [Receive](#receive) portion of the **Settings** page to the folder from where files should be downloaded. Use a comma-delimited list to specify multiple folders. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the Advanced tab. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. After configuration is complete, files are downloaded according to the **Receive** [Automation](#automation-settings) settings, or you can manually click the **Receive** button on the **Transactions** tab. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# CData Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata Connect Arc to CData for data integration and synchronization The CData connector allows you to integrate CData into your data flow by pushing or pulling data from CData. ## Key Capabilities * Support for 200+ data sources through CData drivers (ADO.NET and JDBC) * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation * Caching for more performant queries * Ability to only process new or changed records * Flexible output formats (such as CSV and XML) Follow the steps below to connect Arc to CData. ## Installing the CData Driver You must install the ADO.NET provider (.NET edition) or JDBC driver (Cross-Platform edition) for the appropriate data source before connecting to the data source. > [!NOTE] > > Data sources referenced in the CData connector that require licensing should be licensed on the machine outside of Arc. > [!TIP] > > For step-by-step instructions on downloading and installing CData drivers, see the [CData driver documentation](https://www.cdata.com/kb/help/). ### ADO.NET Follow these steps to download and run the CData installer for an ADO.NET provider: 1. Find the the desired data source in the [list of CData data sources](https://www.cdata.com/drivers/). Click the tile for a data source to bring up the page for that data source. 2. Find the top toolbar that lists the available driver types and click the **ADO.NET** tile. 3. Click **Download** on the resulting page. 4. Download the trial version (30 days of full functionality) or the full provider (requires a purchased license). Contact the Arc licensing team at **[arcsales@cdata.com](mailto:arcsales@cdata.com)** to arrange for provider licensing. 5. Run the downloaded installer. 6. Restart the Arc server to allow the CData connector to detect the new provider. ### JDBC For the CData connector to recognize a JDBC driver, you must install it and place it in the `lib` folder of the Java server hosting Arc. 1. Find the desired data source in the [list of CData data sources](https://www.cdata.com/drivers/). Click the tile for a data source to bring up the page for that data source. 2. Find the top toolbar that lists the available driver types and click the **JDBC** tile. 3. Click **Download** on the next page. 4. Download the trial version (30 days of full functionality) or the full driver (requires a purchased license). Choose the download option that best matches your Arc setup (`.exe` for Windows, `.zip` for Linux, or `.dmg` for Mac). Contact the Arc licensing team at **[arcsales@cdata.com](mailto:arcsales@cdata.com)** to arrange for driver licensing. 5. Navigate to the driver installation directory and find the driver `.jar` file and `.lic` (license) file inside the `/lib/` folder. 6. Copy the driver `.jar` and `.lic` files into the `/lib/` folder of the Java server hosting Arc. 7. Restart the Arc server to allow the CData connector to detect the new driver. ## Establish a Connection To allow Arc to use data from CData, you must enter the connection credentials into each instance of the CData connector. Connections to CData can not be reused across multiple CData connectors. Follow these steps to establish a connection: 1. Drag the CData connector into an Arc flow and give it a name. 2. In the **Connection** portion of the **Settings** tab, select your **Data Source** from the drop-down menu. 3. Authentication steps vary depending on the selected data source: * If connection fields appear, enter the requested information in each one, then click **Test Connection** to ensure that Arc can connect to CData. If an error occurs, check all fields and try again. * If a **Connect** button appears, click it to sign into the driver using OAuth authentication. 4. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to CData, you must choose the action that the CData connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./cdata-upsert) | Inserts or updates CData data. By default, if a record already exists in CData, an update is performed on the existing data in CData using the values provided from the input. | End | | [Lookup](./cdata-lookup) | Retrieves values from CData and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from CData. It should be formatted as a SQL query against the CData tables. | Middle | | [Lookup Stored Procedure](./cdata-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./cdata-select) | Retrieves data from CData and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./cdata-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./cdata-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # CData Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-ac Advanced configuration options for the CData connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the CData connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The CData connector models CData tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and CData values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The CData connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into CData. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The CData connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the CData connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by CData. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by CData, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from CData. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure CData connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # CData Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the CData connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the CData connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # CData Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-lookup Retrieve values from CData and insert them into an Arc message in the flow. The Lookup action retrieves values from CData and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the CData connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the CData connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the CData connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the CData connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from CData tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The CData connector can perform Lookup queries against a local cache instead of querying CData directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query CData on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries CData directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to CData.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # CData Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-lsp Use a stored procedure to retrieve values from CData and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the CData connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the CData connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the CData connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# CData Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-select Retrieve data from CData and bring it into Arc. The Select action retrieves data from CData and brings it into CData Arc. ## Tables and Columns If you choose Select for the CData connector, you must select a target table (or tables) from CData. Click the **Add** button above the **Tables** pane. ![Add button above the Tables pane](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_select_tableadd.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=fd2a0a21041ae01110fd066c36723d07) A modal appears and lists all available tables. Select the desired table and click **Add**. ![Modal listing available tables to select from](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_tablelist.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=60f089dec2cb7c941456ace30db249a4) The chosen table appears under the **Tables** pane, and the columns in the table appear under the **Columns** pane. The connector automatically detects values that have special relevance (such as the primary key, foreign keys, and similar) and labels them accordingly. ![Tables pane showing the selected table and Columns pane with all columns listed](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_select_columns.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=c34caad84b195ef21baa8cf7dfdaf422) By default, all columns are selected for inclusion in the action for the CData connector. Exclude individual columns by unchecking them. > [!NOTE] > > You can select more tables by clicking **Add** and repeating this process. See [Child Tables](/26.4/self-hosted/en/connectors/cdata/cdata-ac#child-tables) for more information. ### Selecting from Multiple Tables The CData connector supports Selecting multiple tables with a single action. You can accomplish this in two ways: * Assign extra tables as children of a parent table. See [Child Tables](/26.4/self-hosted/en/connectors/cdata/cdata-ac#child-tables) for more information. * Create multiple Select-configured CData connectors, assign each of them a different table, and connect them all to the same connector in the flow. You can assign separate queries to each child table or individual CData connector as needed. See [Filters](#filters) for more information. ## Filters After you select a table, you can optionally use the **Filter** pane to filter the data that the connector selects. The connector translates your filter rules into query syntax, and the syntax appears in the **Query** pane at the bottom of the connector configuration page. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter Rule To create a filter rule, follow these steps: 1. Click **Add rule**. 2. In the drop-down list for the new rule, choose the column that you want to filter. 3. Two new fields appear. In the new drop-down list, choose the filtering condition. * The available filtering conditions depend on the data type for the column you select. For example, a `long` data type has the condition **Greater Than**, whereas a `string` data type does not. 4. If your filtering condition requires a value, enter that value in the blank value field. If your filtering condition does not require a value (for example, if you choose **Is Null**), the value field disappears. 5. When your filter rule is complete, its background changes from red to white. The image below shows an example of a filter rule on a column named **Author**. With this filter rule in place, the connector only selects values in the **Author** column that contain **Smith**. ![Filter rule on the Author column with Contains Smith condition](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_filter_single.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=860bdc9299c39e67408170416b20e898) ### Filter Groups You can apply multiple filter rules to your connector by clicking **Add rule** and creating more rules. Filter rules are separated by groups, and all filters belong to the same group by default. You can create multiple groups to separate filter rules by clicking **Add group**. Each filter rule in a group interacts with the others based on the options you select at the top of the group: * Check the **NOT** box to invert the filter conditions. For example, in the **Author** filter shown above, checking the **NOT** box instructs the connector to select Author values that do *not* contain **Smith**. * Select either **AND** or **OR** to determine the group logic for multiple filters. * If you select **AND**, every filter rule in the group must be true for the filter conditions to be met. * If you select **OR**, at least one filter rule in the group must be true for the filter conditions to be met. Filter groups have simple parent/child relationships. Each filter rule and filter group that belongs to a parent group exists as a single entity for the filter logic. The screenshot below shows multiple filter groups. ![Multiple filter groups with OR logic at the parent level](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_filter_multiple.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=0cfb05ed39e9acc680bed5f320c67b10) In this example, the parent group is using **OR** logic. This means that if *any* of the following statements is true, the filter condition is met: * The **Author** column contains the value **Smith** * The **Id** column equals **1** or **2** * The **Title** column contains the value **Kitchen** and the **UpdatedAt** column is greater than **400** If this filter condition is met, the connector Selects the data that meets the condition and passes it down the flow. ## Advanced Settings In the **Advanced** section, you can configure settings for using specific columns for processing records and updating columns with specific values when rows are processed successfully. These settings are useful for maintaining and tracking changes in your flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured CData connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a CData Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/cdata_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **CData\_Select** connector pulls data from a CData connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **CData\_Select** then passes down the flow from **CSV1**.
# CData Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/cdata/cdata-upsert Insert or update CData data from an Arc message in the flow. The Upsert action updates or inserts CData data. By default, if a record already exists in CData, an update is performed on the existing data in CData using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. > [!NOTE] > > The CData connector is deprecated. Unlike the other database connectors, it does not support Mapping mode, so an XML Map connector is still required to map columns for the Upsert action. ## Tables and Columns If you choose Upsert for the CData connector, you must select a target table (or tables) from CData. Click the **Add** button above the **Tables** pane. ![The Upsert Configuration panel showing the Tables section with an Add button.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_upsert_tableadd.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=45b47cae541aab5e798e777341ed91dd) A modal appears and lists all available tables. Select the desired table and click **Add**. ![The Add Table modal listing available tables, with the Orders table selected and the Add button at the bottom.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_tablelist.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=60f089dec2cb7c941456ace30db249a4) The chosen table appears under the **Tables** pane, and the columns in the table appear under the **Columns** pane. The connector automatically detects values that have special relevance (such as the primary key and foreign keys) and labels them accordingly. ![The Upsert Configuration panel with the Orders table selected, showing the Columns pane with the Id column's UPSERT toggle set to ON and the UPSERT by field set to Id.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_upsert_columns.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b31a18fcd3fffa9977f3b4058bee296f) By default, all columns are selected for inclusion in the action for the CData connector. Exclude individual columns by unchecking them. > [!NOTE] > > You can select more tables by clicking **Add** and repeating this process. See [Child Tables](/26.4/self-hosted/en/connectors/cdata/cdata-ac#child-tables) for more information. ## UPSERT Query Behavior By default, the **UPSERT** slider in the column mapping is set to **ON**. This means the UPSERT mapping determines whether a record exists in the destination before INSERTing it (if there is no matching record) or UPDATEing it (if the record already exists). ![The Upsert Configuration panel with the UPSERT ON toggle highlighted on the Id column.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_upsert_slider.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=8aa832efd3b112703c476e4f0f4d3002) The query that is used to determine that value is determined by the logic described in the following sections. ### UPSERT Slider Set to ON When the slider is set to **ON**, the query behavior changes based on what you select in the **UPSERT By** and **Perform this query...** radio buttons. #### UPSERT By Id If the key column is present in the input XML, a query is issued to the database to determine if any records exist with the matching key: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column A query is issued to the database to determine if any records exist where the selected **UPSERT By** column contains the matching value, by selecting the primary key of the table: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT By** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query You can also write a custom query to select the UPSERT key. This option allows for the lookup to be performed using more complex logic than just a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Slider Set to OFF When the slider is set to **OFF**, the connector simply inserts all queries into the database. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured CData connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, an XML Map connector, and a CData Upsert connector at the end: ![An example Arc flow showing four connectors in sequence: File\_Receive, CSV1, XMLMap1, and CData\_Upsert.](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/cdata_xmlflow_upsert.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=a0b958a15496e3611d73ba0835852b75) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  4. Data from the **CData\_Upsert** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![The XML Map connector's Mapping interface showing the Source tree (source.xml) on the left and the Destination tree (destination.xml) on the right, with no mappings yet drawn between elements.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_xmlmap_upsert.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=0a16821dfd319da0de252249f8a9a1ce)
  5. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![The XML Map connector's Mapping interface with elements from the Source tree mapped to the Destination tree, shown as curved lines connecting fields such as first to firstName and last to lastName.](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_xmlmap_elements.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=e160cb65b3524cee799144326bb013f9)
  6. The **CData\_Upsert** connector performs the Upsert according to the mapping and passes the resulting data to CData.
# Copy Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/copy Adds a branching shape to the flow, sending incoming messages to multiple connected connectors simultaneously. The Copy connector adds a branching shape to the flow, sending incoming messages to other connectors. ## Key Capabilities * Creates multiple processing paths from single input messages * Visual flow branching with unlimited output connections * Maintains original message content across all output paths ## Overview Copy connectors are primarily configured visually in the flow designer. When you connect a Copy connector to other connectors in the flow, messages that reach the Copy connector are sent to all connected connectors. This allows files to be processed by multiple flows. When you add a Copy connector, it automatically creates two output nodes. To add more, click **Add Copy**. If an output node does not have a connection associated with it, messages remain in the Output folder for that node. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Logging | Setting | Description | | - | - | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # CSV Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/csv Converts comma-separated values (CSV) files into XML and generates CSV files from XML, enabling bidirectional data transformation within flows. The CSV connector converts comma-separated values (CSV) files into XML and generates CSV files from XML. ## Key Capabilities * Bidirectional CSV/XML conversion with configurable delimiters and encoding * Header detection and custom column mapping capabilities * Support for various character encodings and special character handling ## Overview XML is the primary format that CData Arc uses to manipulate data within a flow. Therefore, it is useful to convert CSV files into XML as a staging step for further processing in the flow, or to convert XML to a CSV file after the XML has been manipulated. Both of these operations can be accomplished with the CSV connector. For details on converting CSV into XML, see the [Converting CSV to XML](#converting-csv-to-xml) section. For details on converting XML into CSV, see the [Converting XML to CSV](#converting-xml-to-csv) section. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the connector Id and description.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Connector Settings *Settings related to the core operation of the connector.*
SettingDescription
**Column headers present** Whether the CSV file contains a row of headers providing names or context to the values in the file.
**Record Name** The name of elements representing a row in the CSV file when converting to XML. You can use the following macros: %ConnectorID%, %FilenameNoExt%, %RegexFilename:%, and %Header:%. See [Converting CSV to XML](#converting-csv-to-xml) for more details.
#### Other Settings *Settings not included in the previous categories.*
SettingDescription
**Local File Scheme** A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros).
**Line Separator** The character sequence to use to denote line endings. By default, this is determined by the system, but you can use this setting to override the system behavior. Use `CR` or `\r` to denote carriage return, and `LF` or `\n` to denote line feed.
### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**.
  1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**.
  2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**.
  3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**.
### Advanced Tab
SettingDescription
**Processing Delay** The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.*
SettingDescription
**Send** Whether messages arriving at the connector are automatically processed.
#### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting CSV to XML When a CSV file is transformed into XML, the resulting XML has the following structure: ```xml theme={null} ``` Each row (record) in the original file becomes a child of the root element `Items`. The name of all record elements is determined by the **Record Name** option set in [Connector Settings](#connector-settings). Each record element then has child elements corresponding to the values in each row of the input file. Some CSV files include a line of header information that provides context to the values in the file. When the **First line is header information** setting is enabled, this header line is parsed, and the parsed headers are used as the element names for the value elements (the children of the record elements). Otherwise, the value elements are given generic names such as `field_0`, `field_1`, and so on. ## Converting XML to CSV To convert XML to a CSV file, the input XML must have a 'flat' structure. This means that, disregarding the `Items` root element, the depth of the XML structure is two. For example: ```xml theme={null} Citizen Kane 1941 119 Sharknado 2013 86 ``` The XML is interpreted as follows: * Children of the root element are treated as records (rows) in the resulting file * Children of each record element are treated as the values in each row If the **First line is header information** option is enabled, a header row is inserted into the resulting CSV file with the names of each value element to provide context to the values. In the example above, this header row would consist of `title`, `year`, and `runtime`. ## Print View When you view a CSV document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![](/26.4/images/csv_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## CSV Transformation: Using the XML Map Connector Many data transformation flows use the CSV connector in conjunction with the [XML Map Connector](/26.4/self-hosted/en/connectors/xml-map/xml-map). Often, data enters an Arc flow in CSV format and should exit the flow in some other format (for example, a database insert, an EDI file, or an insert into a CRM or ERP data source), or vice versa. Arc uses a single streamlined approach to these data transformation requirements:
  • Model the input format as XML
  • Model the output format as XML
  • Use the XML Map connector to map between the input XML and the output XML
Therefore, the CSV connector is commonly adjacent to an XML Map connector in the flow:
  • When CSV files are the *input* to the flow, the CSV connector converts a CSV file to XML and then passes that XML off to the XML Map connector to be transformed
  • When CSV files are the *output* from the flow, the CSV connector receives XML from the XML Map connector and converts it into a CSV file
An XML Map connector requires a sample XML structure for both the mapping input/source and the mapping output/destination. Use the [Sample Files](#sample-files-tab) tab to upload a sample file. Next, connect a CSV connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the CSV connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > The structure of this test file should be representative of future files. In other words, all of the CSV files processed by the CSV connector (and then the XML Map connector) should have the same columns as your test file. You might need to set up multiple CSV connectors and multiple XML Map connectors to handle distinct CSV structures. ## CSV Operations In addition to the [Operations](/26.4/self-hosted/en/scripting/operations/operations) provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except for two details: 1. They must be called through the `connector.rsc` endpoint. 2. They must include an auth token. For example, calling a connector operation using both of these rules might look something like this: ```xml theme={null} ``` Operations specific to the functionality of the CSV connector are listed below. ### csvListRecords Loops over every record in a specified CSV file or string. See [Functions](../scripting/value-formatters/function-formatters) for more details. #### Required Parameters * **file**: The path to the CSV file. #### Optional Parameters * **data**: If the CSV data exists as a string rather than stored in a CSV file, use this parameter instead of **file**. * **columns**: The comma-separated list of columns to include in the output (if unspecified, all columns are included). * **requireheader**: By default, the first row of data is interpreted as column headers; pass *false* to this parameter to use generic column names (for example, c1, c2, c3). #### Output Any script in a *csvListRecords* operation executes multiple times: once for each record/row in the input CSV file/data. Within the operation, individual CSV values are accessible using the *csv* formatter. This formatter takes a column name as a parameter, and outputs the value in that column for the current record. For example, imagine the CSV input data contains a set of items purchased in an order, and the name of the item is held in the `ItemName` column. The following script generates XML containing each `ItemName` value in an `Item` element: ```xml theme={null} [csv('ItemName')] ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Database Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database-connectors Arc Database Connectors support connecting to a variety of databases for reading, writing, and upserting data within your flows. CData Arc Database Connectors support connecting to a variety of databases. Select the name of a connector from the navigation pane to access documentation for that connector. ## Video Resources Watch this short video to learn more about back-end integration with applications and databases in Arc. This is part two of a three-part video series that covers each step of an end-to-end B2B integration: managed file transfer (using AS2 as an example), back-end integration, and EDI translation and mapping. Links to the other parts of the series are below. [https://www.youtube.com/embed/wk92g8\_Jmn0](https://www.youtube.com/embed/wk92g8_Jmn0) * Part One: [AS2 Setup in Minutes with CData Arc](https://youtu.be/YjMpP70fczo) * Part Three: [EDI Mapping and Integration in CData Arc](https://youtu.be/wk92g8_Jmn0) # Database Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database Connect Arc to Database for data integration and synchronization The Database connector allows for connections to many different database types, and can push or pull data from these databases. ## Key Capabilities * Universal database connectivity supporting major database platforms * Support for both ADO.NET and JDBC drivers * Comprehensive transaction management with commit and rollback capabilities * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation * Advanced query customization and parameter binding Follow the steps below to connect Arc to Database. ## Establish Database Connection Before you can add the Database connector to your flow, you must configure the connection to your database on the local machine. > [!NOTE] > > Data sources referenced in the Database connector that require licensing should be licensed on the machine outside of Arc. ### ADO.NET The [.NET edition](/26.4/self-hosted/en/getting-started/installation/windows-edition) of Arc primarily uses ADO.NET drivers. In order for an ADO.NET provider to be visible to Arc, it must be defined in the machine.config file and registered in the Global Assembly Cache (GAC). This can be accomplished manually by knowledgeable users, but CData recommends you install the database client on the machine hosting Arc, because the database-specific installation wizard handles defining and registering the provider. ### JDBC In order for a JDBC driver to be visible to Arc, it must be included as part of the Java classpath. Arc includes drivers for H2, MySQL, PostgreSQL, and SQL Server; refer to the documentation for the Java web servlet to add a different driver to the classpath. > [!NOTE] > > The Database connector supports JDBC drivers for databases other than the default options. However, custom drivers might require overwriting the queries issued by the connector to fit the syntax expected by the database. Queries can be overwritten using the *Write custom query* section of an Input Mapping or Output Mapping. ## Configure Database Connector After you configure the database connection, you can add a Database connector to your Arc flow. You must enter the database connection credentials into each instance of the Database connector. Follow these steps to establish a connection: 1. Drag the Database connector into an Arc flow and give it a name. 2. In the **Database Connection** portion of the **Settings** tab, choose your **Driver Name** from the list. The **Driver Class** is automatically populated. 3. Choose your **Settings Format**. 1. If you choose **Property List**, complete the driver-specific properties. These vary by driver. 2. If you choose **Connection String** or **DatabaseURL**, supply the full connection string to the database. 4. Click **Test Connection** to ensure that Arc can connect to the Database. If an error occurs, check all fields and try again. 5. Click **Save Changes**. ### Other Database Connections The **Database Connection** dropdown includes an **Other** option. When you choose **Other**, a **Connection Type** option appears, asking you to select **ADO.NET** or **ODBC**. * If you choose **ADO.NET**, enter your **Driver Class**, **Data Source**, **User**, and **Password** details. * If you choose **ODBC**, ensure you have followed the instructions below to create a system DSN, then select it in **Data Source Name**. Complete the **Driver**, **User**, and **Password** details. #### ODBC A data source name (DSN) is required to connect to a database via an ODBC driver. Create it with the Microsoft ODBC Administrator. CData must use a system DSN connection for the ODBC driver. Once you have created the DSN, select it in the **Data Source Name** connector dropdown. The Database connector supports ODBC drivers for databases other than the default options. However, custom drivers might require overwriting the queries issued by the connector to fit the syntax expected by the database. See [Pre-Processing and Post-Processing Queries](./database-ac#pre-processing-and-post-processing-queries) for more information. You can also use the **Write custom queries** option on the [Select](./database-select) action. ## Select an Action After establishing a connection to Database, you must choose the action that the Database connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./database-upsert) | Inserts or updates Database data. By default, if a record already exists in Database, an update is performed on the existing data in Database using the values provided from the input. | End | | [Lookup](./database-lookup) | Retrieves values from Database and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Database. It should be formatted as a SQL query against the Database tables. | Middle | | [Lookup Stored Procedure](./database-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./database-select) | Retrieves data from Database and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./database-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./database-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Database Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-lookup Retrieve values from the database and insert them into an Arc message in the flow. The Lookup action retrieves values from Database and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Database connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Database connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Database connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Database connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Database tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Database connector can perform Lookup queries against a local cache instead of querying Database directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Database on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Database directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Database.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Database Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-lsp Treat data coming into the connector as input for a stored procedure, then insert the result into an existing Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Database connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Database connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Database connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Database Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-upsert Insert or update data in the database from an Arc message in the flow. The Upsert action updates or inserts Database data. By default, if a record already exists in Database, an update is performed on the existing data in Database using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Database Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Database. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/database/database-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Database connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Database Upsert connector at the end: ![](/26.4/images/database_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Database\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Database.
# Discard Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/discard A terminal connector that removes all received messages with no output, used as the final step in a flow to intentionally drop unwanted messages. The Discard connector can only be used as the last connector in a [flow](../flows/flows). When it is connected to another connector, any messages passed into the connector are discarded. ## Key Capabilities * Terminal message disposal—discards all received messages with no output or side effects * Guaranteed end-of-flow enforcement, ensuring messages are not accidentally passed to further connectors * Lightweight and configuration-free, requiring no setup to drop unwanted messages from a flow ## Overview The Discard connector is a terminal connector that removes any messages it receives, producing no output. Use it as the final step in a flow to intentionally drop messages that require no further processing or routing. ## Connector Configuration ### Settings Tab *Settings related to the core configuration of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. # Dropbox Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dropbox Configuration and usage guide for the CData Arc Dropbox connector, which supports uploading to and downloading from the Dropbox cloud storage service. The Dropbox connector supports uploading to and downloading from the Dropbox cloud storage service. ## Key Capabilities * OAuth-based cloud storage integration with automatic token refresh * Upload and download from the connected Dropbox account with recursive subdirectory support * File filtering, deletion management, and overwrite protection * Comprehensive proxy support for enterprise network environments * Optional caching to ensure only new or updated files are downloaded ## Overview Each Dropbox connector connects to a specified Dropbox storage account. Within the remote storage, you set a single folder as the upload target, and a list of folders as the download target. Files that reach the Dropbox connector in the CData Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specific file names and/or file extensions. Account authentication uses OAuth, so you must enter account credentials in an external site portal the first time the connector connects. The external portal generates an OAuth token that the connector uses to connect, then the connector handles token refresh automatically. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
Use the **Connect** button to authenticate the connector with the storage service the first time. You are redirected to a site login portal. After you provide the account credentials, an authentication token is returned to Arc and stored for repeat use. #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | Files processed by the connector are uploaded to this folder. | | **Overwrite** | Whether to overwrite or skip existing files. | | **Namespace Id** | Points the connector to a Team folder in Dropbox Business. Leave blank if you are not connecting to a Team folder. | #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | **File Mask** | A glob pattern that determines which files within the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse remote subdirectories** | Whether to download files in subfolders of the target remote path. | | **Timeout** | The length of time (in seconds) the server waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send Automation** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection To establish a connection with the Dropbox connector, use the **Connect** button on the [Settings tab](#settings-tab) to be directed to a login portal site. Enter your account credentials in this portal; it returns an authentication token to the connector. This process is only required the first time the connector connects to the storage service. ## Uploading and Downloading ### Upload To upload files, set the **Remote Folder** in the [Send](#send) portion of the **Settings** page to the folder where files should be uploaded. Each Dropbox connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the container should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Download To download files, set the **Remote Folder** in the [Receive](#receive) portion of the **Settings** page to the folder from where files should be downloaded. Use a comma-delimited list to specify multiple folders. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the Advanced tab. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# EANCOM Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/eancom Configuration and usage guide for the CData Arc EANCOM connector. EANCOM connectors support generating EANCOM documents (also known as GS1 documents) from XML or converting EANCOM documents into XML. ## Key Capabilities * Full bidirectional translation between EANCOM and XML formats * Comprehensive interchange header validation and generation * Support for multiple EANCOM versions with automatic schema detection * Interchange and functional acknowledgment support ## Overview When receiving EANCOM documents, EANCOM connectors validate interchange headers and convert the EANCOM document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data within a flow. The EANCOM connector automatically reads the input file to determine the appropriate schema, then parses the document according to this schema. When generating EANCOM documents, EANCOM connectors convert XML into EANCOM document syntax and apply the appropriate interchange headers. This is useful as the final step for creating an EANCOM document, after the XML data has been fetched and transformed elsewhere in the flow. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert EANCOM documents into XML, or generate EANCOM documents from XML. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner Identifier (UNB2.1)** | The unique Id identifying the sending party in the EANCOM communication. When you are generating an EANCOM document, this should be your identifier. Only applicable when **Dynamically process partners** is unchecked. | | **Recipient Identifier (UNB3.1)** | The unique Id identifying the receiving party in the EANCOM communication. When you are generating an EANCOM document, this should be your trading partner's identifier. Only applicable when **Dynamically process partners** is unchecked. | #### Interchange Settings *Settings related to the EANCOM interchange headers. When converting EANCOM documents into XML, the document headers are validated against these settings; when generating EANCOM documents from XML, these settings are used to generate document headers.* | Setting | Description | | - | - | | **Syntax Identifier (UNB1.1)** | Identifies the character set used in the EANCOM document. | | **Syntax Version (UNB1.2)** | In combination with the **Syntax Identifier**, determines the syntax to be used in the EANCOM document. Based on your selection here, other interchange setting options appear or disappear on the [Advanced tab](#advanced-tab). | | **Partner Code Qualifier (UNB2.2)** | The qualifier for the **Sender Identifier**, which provides context to the value (for example, an EAN location number). | | **Personal Code Qualifier (UNB3.2)** | The qualifier for the **Recipient Identifier**, which provides context to the value (for example, an EAN location number). | | **Test Indicator (UNB11)** | Whether the interchange is in test mode or production mode. If you check this, interchange headers are *not* validated when documents are received. | | **Functional Group** | Check this to automatically add the sender and recipient identifiers to the [Functional Group Settings](#functional-group-settings) on the **Advanced** tab. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **Technical acknowledgment (CONTRL)** | Whether a technical CONTRL ACK should be returned (when receiving) and requested (when sending). A technical acknowledgment serves as a receipt of the interchange. | | **Functional acknowledgment (CONTRL)** | Whether a functional CONTRL ACK should be returned (when receiving) and requested (when sending). A functional acknowledgment serves as an indication of acceptance or rejection of the received interchange. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, etc.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Repetition Char** | The character that indicates repetition of element values. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Interchange Settings *Additional settings related to the EANCOM interchange headers. These options appear or disappear based on the **Syntax Version** specified on the Settings tab.* | Setting | Description | | - | - | | **Service Code List Directory Version Number (UNB1.3)** | Further specifies the syntax to be used in the EANCOM document. Only applicable for EANCOM syntax version 4. | | **Character Encoding (UNB1.4)** | Specifies the character encoding (such as ASCII or UTF-8). Only applicable for EANCOM syntax version 4. | | **Address for Reverse Routing (UNB2.3)** | The optional address in the sender's system to which responding interchanges should be sent. Only applicable for EANCOM syntax versions prior to version 4. | | **Partner Internal ID (UNB2.3)** | An additional sender identifier to facilitate internal routing of response interchanges. Only applicable for EANCOM syntax version 4. | | **Partner Internal Sub-Identification (UNB2.4)** | Further identifies the sender when sub-level identification is required. Only applicable for EANCOM syntax version 4. | | **Routing Address (UNB3.3)** | The optional address in the recipient's system to which interchanges should be routed. Only applicable for EANCOM syntax versions prior to version 4. | | **Personal Internal ID (UNB3.3)** | An additional recipient identifier to facilitate internal routing of received interchanges. Only applicable for EANCOM syntax version 4. | | **Personal Internal Sub-Identification (UNB3.4)** | Further identifies the recipient when sub-level identification is required. Only applicable for EANCOM syntax version 4. | | **Partner Password (UNB6.1)** | Reference or password to gain access to the recipient's system. | | **Partner Password Qualifier (UNB6.2)** | The qualifier that provides context to the **Partner Password**, if applicable. | | **Application Reference ID (UNB7)** | Identifies the application group to which the messages in the interchange relate. | | **Processing Priority Code (UNB8)** | Code for requesting processing priority for the interchange. | | **Communication Agreement (UNB10)** | Defines the type of communication agreement controlling the interchange. | #### Functional Group Settings *Settings related to the functional group headers of EANCOM documents. These optional identifiers can help group similar interchanges together, or facilitate sub-addressing within an organization.* | Setting | Description | | - | - | | **Application Partner Identifier (UNG2.1)** | Identifies the application sending the document (such as a division, branch, or computer system). | | **Application Personal Identifier (UNG2.1)** | Identifies the application for which the document is intended. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An interchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the interchange. When checked, the connector groups all transactions into a single output file. | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing an EDI qualifier include child elements containing the qualifier code and value. For example: `STShip To`. | | **Functional Acks** | By default, all functional acknowledgments (997, 999) are routed to the connector selected in the flow diagram, and XML translations are not received in the **Transactions** tab. Check this to have the translated acknowledgment also be included in the **Transactions** tab. This allows functional acknowledgments to be integrated into a destination source in addition to new EDI documents. | | **Generate Description As** | When translating EANCOM into XML, descriptions of the EANCOM segments and elements can be provided as context for the EANCOM data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects EDI structures that have hierarchical relationships embedded in the EDI data, and generates XML with these hierarchical relationships represented as parent-child relationships. See [Master-Detail Hierarchy: Translating CPS Loops](#master-detail-hierarchy-translating-cps-loops) for more information. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Track UNB2.1** | Whether to add UNB2.1 values as tracked headers to processed messages. | | **Track UNB3.1** | Whether to add UNB3.1 values as tracked headers to processed messages. | | **Track Transaction Types** | Whether to add transaction types as tracked headers to processed messages. | | **Validate Identifiers** | Check this to ensure that the identifiers in the translated document match the identifiers in the connector's configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_eancom_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_shared_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## XML to EANCOM When generating EANCOM files, select **XML to EANCOM** as the **Translation Type**. The application reads the header information for the interchange, functional group, and message sections from the settings configured on the [Settings tab](#settings-tab). These settings include the sender identifier and code qualifier, receiver identifier and code qualifier, and so on. Once you have configured the connector with the settings from your trading partner agreement, click the **Transactions** tab and select **More > Create Test Files**. The application creates a set of test XML files that provide example data of what the XML should look like for the connector to generate an EANCOM document. You can also select the documents and click **Send** to see the generated EANCOM documents. ## EANCOM to XML When generating XML, select **EANCOM to XML** as the **Translation Type**. The application only processes EANCOM files when this option is selected. You need to configure the settings for the trading partner agreement on this page for the application to validate the information in EANCOM documents during processing. XML files are generated into the Output folder for this connector. Click the **Transactions** tab and select **More > Create Test Files** to create test EANCOM files for several EANCOM documents. You can select these documents and click **Send** to see the generated XML. ## Use With the XML Map Connector In accordance with Arc convention, the EANCOM connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound EANCOM documents, or as the following step in the flow when receiving inbound EANCOM documents. The interaction between the XML Map connector and the EANCOM connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the EANCOM connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the EANCOM connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the EANCOM document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## Acknowledgements Acknowledgments are routed to the connector specified by the grey dotted line in the flow diagram, as shown in the following image: ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/edi_ack_flow.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=f1100a5d688c43c8014d3a70587bedf9) The specified connector generates acknowledgments using the transaction data received by the current connector. The specified connector adds interchange headers and passes the acknowledgment along in the flow like any other message. Thus this should typically be set to a pre-configured EANCOM connector that generates outbound documents for the intended recipient. Acknowledgments can be automatically generated and processed by the application. There are two commonly requested types of acknowledgments used in EANCOM communications, interchange and functional. Both of the acknowledgments types are classified as EANCOM CONTRL messages. When EDI messages that require an acknowledgement have not received that acknowledgement, a Pending-Timeout header is added to the message. The message has a *Pending* status while it waits for its ACK. If it does not receive the ACK within the allotted time, the Pending-Timeout header is populated with the date and time when the message status changed to *Error*. When the ACK is received and the message is updated to *Success*, a new header is added with an empty string value. #### Interchange Acknowledgments This is an indication that an interchange has taken place between the two parties (though not necessarily that any individual message has been exchanged). They are automatically generated in response to a request in the interchange. #### Functional Acknowledgments Functional acknowledgments are an indication that an individual message such as an Invoice or PO has been accepted. These are generated in response to a bidirectional agreement by both parties. ### Message Originator #### Requesting Acknowledgements EANCOM connectors in XML-to-EDI mode can be configured so that both interchange and functional acknowledgments can be requested for a message. Select the **Technical acknowledgment (CONTRL)** and **Functional acknowledgment (CONTRL)** checkboxes on the [Settings tab](#settings-tab) to direct the connector to maintain a Pending ACK status for the transmission until the appropriate acknowledgments have been returned and processed. ![Receiving ACKs](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-receive.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=e753246ca35494ac86549872227df731) The illustration above shows a complete message exchange from the standpoint of the message originator. An EANCOM connector in XML-to-EDI mode generates the document to be exchanged **(1)** and holds it in a Pending ACK state. The trading partner processes the transmission in their business logic and creates acknowledgments in accordance with the configured exchange parameters and the bidirectional agreement between parties **(2)**. When the acknowledgments are returned, they are parsed with and an EANCOM connector in EDI-to-XML mode and routed to the original EANCOM connector (XML-to-EDI) to resolve the status of the transfer **(3)**. #### Processing Acknowledgements When interchange and functional EANCOM acknowledgments are received at an EANCOM connector in EDI-to-XML mode, the connector can be configured to automatically route any detected acknowledgments to the EANCOM connector that first generated the transmission. In the flow diagram, drag the grey circle from the receiving (EDI-to-XML) EANCOM connector onto the originating (XML-to-EDI) EANCOM connector. The configured (XML-to-EDI) EANCOM connector pairs the acknowledgments to the original message and resolves the matching message. ### Message Recipient #### Generating Acknowledgements When an EANCOM connector in EDI-to-XML mode processes received messages from the originator and generates XML, tell the connector to automatically generate CONTRL acknowledgments by selecting the **Acknowledgment** check boxes in the Settings tab. Route these acknowledgments to an EANCOM connector in XML-to-EDI mode to generate the EANCOM acknowledgement. In the flow diagram, drag the grey circle onto the (XML-to-EDI) EANCOM connector that generates EANCOM documents for this partner. ![Generate ACKs](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-create.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=3d2adfeb1f049885a0cf400ccc9b6cb5) As shown in the diagram above, when a partner sends a message where an acknowledgment is expected **(1)**, the (EDI-to-XML) EANCOM connector configured for that partner automatically generates an acknowledgment **(2)** as an XML file containing the transactional information relevant for that file. This XML acknowledgment must be routed back to an (XML-to-EDI) EANCOM connector **(3)** so that all of the EDI party agreement settings are used when generating the EANCOM file that is sent back to your trading partner. ## Master-Detail Hierarchy: Translating CPS Loops In EDI documents, most hierarchical relationships are represented by the order of EDI segments. Some EDI structures, like CPS Loops (which are found in DESADV documents), do not follow this convention and instead have the hierarchical relationships embedded in the EDI element data itself. This can make it difficult to preserve these hierarchical relationships when converting the EDI data to XML. The EANCOM connector supports preserving hierarchical relationships in CPS segments via the **Nest Loops** setting on the [Advanced tab](#advanced-tab). When enabled, the connector parses the elements in the CPS segments to determine which segments "belong to" other segments in a hierarchical relationship. These hierarchical relationships are reflected in the output XML as parent-child relationships: in other words, this setting converts hierarchy implied by the EDI content into hierarchy represented by the XML structure. This section briefly explains both how hierarchy is encoded in CPS data, and how this hierarchy is converted into XML when **Nest Loops** is enabled. ### CPS Hierarchy All CPS segments have two values that help establish hierarchy: * an Id value, which identifies the current CPS segment (this value is stored in CPS01, or the first element in a CPS segment) * a parent-Id value, which identifies the current CPS segment's hierarchical parent (this value is stored in CPS02, or the second element in a CPS segment) For example, a CPS segment has an Id value of '2'. If the next CPS segment should "belong to" this prior segment in a hierarchical relationship, the next CPS segment's parent-Id should also be '2'. If a CPS segment's parent-ID is '0', the segment does not have a parent (it is at the top level of the hierarchy). ### Converting CPS Hierarchy to XML When **Nest Loops** is enabled, the EANCOM connector handles the conversion of CPS hierarchy to XML hierarchy automatically. The connector parses the Id and parent-Id values from CPS segments and ensures that the resulting XML elements are appropriately nested (indented) inside the elements representing their parents. In other words, if `segmentA` has a parent-Id value that is equal to `segmentB`, the resulting XML has `segmentA` as a child of `segmentB`. In this way, the hierarchical relationships are preserved in the XML structure when the EDI data is translated. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# EDI Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/edi-connectors CData Arc EDI connectors convert a variety of EDI business document standards to and from XML for back-end integration and translation workflows. Arc Electronic Data Interchange (EDI) connectors support converting a variety of EDI business document standards to and from XML. Arc uses XML as the primary format for data manipulation and transformation, so converting files between EDI and XML helps streamline EDI processing by: * Integrating EDI documents into a back-end system (such as a database or a cloud application) * Translating EDI documents into flat file formats (such as CSV or TSV) * Generating EDI documents from external data EDI documents are parsed according to standard schemas provided with the application. If necessary, additional schemas can be provided to the EDI connectors by adding the schema files to a connector's *Schemas* folder. The EDI connectors also support validating EDI transaction headers to ensure that documents are sent and received by the expected parties, and the connectors can automatically generate EDI acknowledgments to provide non-repudiation assurance. ## Video Resources Watch this video to see an example of EDI mapping and integration in Arc. This is part three of a three-part video series that covers each step of an end-to-end B2B integration: managed file transfer (using AS2 as an example), back-end integration, and EDI translation and mapping. Links to the other parts of the series are below. [https://www.youtube.com/embed/wk92g8\_Jmn0](https://www.youtube.com/embed/wk92g8_Jmn0) * Part One: [AS2 Setup in Minutes with CData Arc](https://youtu.be/YjMpP70fczo) * Part Two: [Back-End Integration in CData Arc—Applications and Databases](https://youtu.be/uhWYwsXRc3Q) # EDIFACT Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/edifact Configuration and usage guide for the CData Arc EDIFACT connector. The EDIFACT connector can generate EDIFACT documents from XML or convert EDIFACT documents into XML. ## Key Capabilities * Full bidirectional translation between EDIFACT and XML formats * Comprehensive interchange header validation and generation * Support for multiple EDIFACT versions with automatic schema detection * Interchange and functional acknowledgment support ## Overview When receiving EDIFACT documents, EDIFACT connectors validate EDIFACT interchange headers and convert the EDIFACT document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The EDIFACT connector automatically reads the input file to determine the appropriate EDIFACT schema, then parses the document according to this schema. When generating EDIFACT documents, EDIFACT connectors convert XML into EDIFACT document syntax and apply the appropriate interchange headers. This is useful as the final step for creating an EDIFACT document, after the XML data has been fetched and transformed elsewhere in the flow. > [!NOTE] > > Interchange header validation can be avoided by enabling the **Test Indicator** setting. An EDIFACT connector can also automatically generate acknowledgments to incoming EDIFACT documents. For more information, see [EDIFACT Acknowledgments](#edifact-acknowledgments). ## Prerequisites Before you begin, make sure you have the correct schema installed for the document type you need to process. The EDIFACT connector natively supports the following schema definitions: 4.1, D03A, D04A, D07A, D96A, D96B, D97A, D99A and D99B. You can find them in the `app_data/edifact_schemas` directory of your Arc installation. If you need different schemas, you can download the additional schema files from the [CData website](https://arc.cdata.com/schemas/) for free. After you download and extract the zip archive, place the entire folder into an appropriately named subdirectory of the `app_data/edifact_schemas` directory. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert EDIFACT documents into XML, or generate EDIFACT documents from XML. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner Identifier (UNB2.1)** | The unique Id identifying the sending party in the EDIFACT communication. When you are generating an EDIFACT document, this should be your identifier. Only applicable when **Dynamically process partners** is unchecked. | | **Personal Identifier (UNB3.1)** | The unique Id identifying the receiving party in the EDIFACT communication. When you are generating an EDIFACT document, this should be your trading partner's identifier. Only applicable when **Dynamically process partners** is unchecked. | #### Interchange Settings *Settings related to the EDIFACT interchange headers. When generating EDIFACT documents from XML, these settings are used to generate document headers. When parsing EDIFACT documents, these settings are used to validate the incoming document.* | Setting | Description | | - | - | | **Syntax Identifier (UNB1.1)** | Identifies the character set used in the EDIFACT document. | | **Syntax Version (UNB1.2)** | In combination with the **Syntax Identifier**, determines the syntax to be used in the EDIFACT document. Based on your selection here, other interchange setting options appear or disappear on the [Advanced tab](#advanced-tab). | | **Partner Code Qualifier (UNB2.2)** | The qualifier for the **Sender Identifier**, which provides context to the value (for example, an EAN location number). | | **Personal Code Qualifier (UNB3.2)** | The qualifier for the **Recipient Identifier**, which provides context to the value (for example, an EAN location number). | | **Test Indicator (UNB11)** | Whether the interchange is in test mode or production mode. If you check this, interchange headers are *not* validated when documents are received. | | **Functional Group** | Check this to automatically add the sender and recipient identifiers to the [Functional Group Settings](#functional-group-settings) on the **Advanced** tab. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **Technical acknowledgment (CONTRL)** | Whether a technical CONTRL ACK should be returned (when receiving) and requested (when sending). A technical acknowledgment serves as a receipt of the interchange. | | **Functional acknowledgment (CONTRL)** | Whether a functional CONTRL ACK should be returned (when receiving) and requested (when sending). A functional acknowledgment serves as an indication of acceptance or rejection of the received interchange. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Repetition Char** | The character that indicates repetition of element values. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Interchange Settings *Additional settings related to the EDIFACT interchange headers. These options appear or disappear based on the **Syntax Version** specified on the Settings tab.* | Setting | Description | | - | - | | **Service Code List Directory Version Number (UNB1.3)** | Further specifies the syntax to be used in the EDIFACT document. Only applicable for EDIFACT syntax version 4. | | **Character Encoding (UNB1.4)** | Specifies how characters are encoded (such as ASCII or UTF-8). Only applicable for EDIFACT syntax version 4. | | **Address for Reverse Routing (UNB2.3)** | The optional address in the sender's system to which responding interchanges should be sent. Only applicable for EDIFACT syntax versions prior to version 4. | | **Partner Internal ID (UNB2.3)** | An additional sender identifier to facilitate internal routing of response interchanges. Only applicable for EDIFACT syntax version 4. | | **Partner Internal Sub-Identification (UNB2.4)** | Further identifies the sender when sub-level identification is required. Only applicable for EDIFACT syntax version 4. | | **Routing Address (UNB3.3)** | The optional address in the recipient's system to which interchanges should be routed. Only applicable for EDIFACT syntax versions prior to version 4. | | **Personal Internal Identification (UNB3.3)** | An additional recipient identifier to facilitate internal routing of received interchanges. Only applicable for EDIFACT syntax version 4. | | **Personal Internal Sub-Identification (UNB3.4)** | Further identifies the recipient, for when sub-level identification is required. Only applicable for EDIFACT syntax version 4. | | **Partner Password (UNB6.1)** | Reference or password to gain access to the recipient's system. | | **Partner Password Qualifier (UNB6.2)** | The qualifier that provides context to the **Partner Password**, if applicable. | | **Application Reference ID (UNB7)** | Identifies the application group to which the messages in the interchange relate. | | **Processing Priority Code (UNB8)** | Code for requesting processing priority for the interchange. | | **Communication Agreement (UNB10)** | Defines the type of communication agreement controlling the interchange. | #### Functional Group Settings *Settings related to the functional group headers of EDIFACT documents. These optional identifiers can help group similar interchanges together or facilitate sub-addressing within an organization.* | Setting | Description | | - | - | | **Application Partner Identifier (UNG2.1)** | Identifies the application sending the document (such as a division, branch, or computer system). | | **Application Personal Identifier (UNG2.1)** | Identifies the application for which the document is intended. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An interchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the interchange. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is EDI to XML. | | **Use Reference Id** | When checked, when you are translating EDIFACT into XML, reference Ids are used to name XML elements. Only applicable when the **Translation Type** is EDI to XML. For example: `<_143>value`. If this is unchecked, the Reference designator is used to name XML elements: `value` | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing an EDI qualifier include child elements containing the qualifier code and value. For example: ``   `ST`   `Ship To` `` | | **Functional Acks** | By default, all functional acknowledgments (997, 999) are routed to the connector selected in the flow diagram, and XML translations are not received in the **Transactions** tab. Check this to have the translated acknowledgment also be included in the **Transactions** tab. This allows functional acknowledgments to be integrated into a destination source in addition to new EDI documents. | | **Generate Description As** | When translating EDIFACT into XML, descriptions of the EDIFACT segments and elements can be provided as context for the EDIFACT data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects EDI structures that have hierarchical relationships embedded in the EDI data, and generates XML with these hierarchical relationships represented as parent-child relationships. See [Master-Detail Hierarchy: Translating CPS and HYN Loops](#master-detail-hierarchy-translating-cps-and-hyn-loops) for more information. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Track UNB2.1** | Whether to add UNB2.1 values as tracked headers to processed messages. | | **Track UNB3.1** | Whether to add UNB3.1 values as tracked headers to processed messages. | | **Track Transaction Types** | Whether to add transaction types as tracked headers to processed messages. | | **Validate Identifiers** | Check this to ensure that the identifiers in the translated document match the identifiers in the connector's configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_edifact_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_shared_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting Formats The following sections detail the process of converting EDIFACT to XML and vice versa. ### EDIFACT to XML Setting the **Translation Type** to *EDI to XML* instructs the connector to parse incoming EDIFACT documents into XML. The connector first reads all of the header information for the interchange and functional group sections of the document and validates them against the configured connector settings (unless **Test Indicator** is checked). The connector then parses out the specific EDIFACT schema used in the document and loads the schema from the `edifact_schemas` folder on disk (additional schema files can be [downloaded from our website](https://arc.cdata.com/schemas/) for free). Using the schema, the connector generates XML representing the EDIFACT structure of the document, populates the XML with the values from the document, and provides context to each value either as XML comments or as attributes in the XML elements (based on the value of **Generate Descriptions As**). To see this process with a set of test EDIFACT documents, navigate to the **Transactions** tab of an EDIFACT connector (with **Translation Type** set to *EDI to XML*) and select **More > Create Test Files**. EDIFACT documents for an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting XML. Once EDIFACT documents are converted to XML, the data can be transformed and manipulated in many ways. Commonly, EDIFACT data needs to be stored in a database or other back-end application system. Since Arc uses XML to represent `inserts` into these back-end systems, storing the EDIFACT data becomes a matter of mapping one XML structure onto another. This is typically done with the visual designer-driven [XML Map connector](./xml-map/xml-map). ### XML to EDIFACT Setting the **Translation Type** to *XML to EDI* instructs the connector to generate an EDIFACT document out of an XML representation of the document. After the connector has constructed the EDIFACT message out of the data parsed from the XML, it adds functional group and interchange headers according to the configured connector settings. To see this process with a set of test XML files, navigate to the **Transactions** tab of an EDIFACT connector (with **Translation Type** set to *XML to EDI*) and select **More > Create Test Files**. XML files representing an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting EDIFACT document. ### Print View When you view an EDIFACT document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![EDIFACT print preview](/26.4/images/edifact_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## Use With the XML Map Connector In accordance with Arc convention, the EDIFACT connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound EDIFACT documents, or as the following step in the flow when receiving inbound EDIFACT documents. The interaction between the XML Map connector and the EDIFACT connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the EDIFACT connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the EDIFACT connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the EDIFACT document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## EDIFACT Acknowledgments The following sections detail the two types of EDIFACT acknowledgments (ACKs), and how Arc expects, processes, and generates ACKs. ### Technical versus Functional Acknowledgments Technical ACKs, sometimes called interchange ACKs, are an indication that an interchange has taken place between the two parties, although not necessarily that any individual message has been exchanged. These serve as a receipt to the sender indicating that the EDIFACT message was successfully received, but it does not specify whether there were any issues with processing the message content. Functional ACKs are an indication that an interchange has been processed by the receiving party. They can report acceptance, acceptance with issues, or rejection of the received document. These serve as both a receipt that an interchange was successfully received and that it was fully processed. ### Expecting, Processing, and Generating Acknowledgments #### Expecting Acknowledgements EDIFACT connectors operating in XML to EDI mode can be configured so that technical and/or functional ACKs are expected for a message. When one or both **Technical acknowledgment (CONTRL)** and **Functional acknowledgment (CONTRL)** are checked in the [Acknowledgments](#acknowledgments) section of the Settings tab, the connector maintains a Pending ACK status for a transmission until the appropriate ACKs have been returned and processed. This means that the connector status can be used to determine whether the recipient has confirmed that they received the interchange. When EDI messages that require an acknowledgement have not received that acknowledgement, a Pending-Timeout header is added to the message. The message has a *Pending* status while it waits for its ACK. If it does not receive the ACK within the allotted time, the Pending-Timeout header is populated with the date and time when the message status changed to *Error*. When the ACK is received and the message is updated to *Success*, a new header is added with an empty string value. The following diagram shows the logical flow involved when Arc expects an ACK from an invoice document: ![EDIFACT Receive ACK](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-receive.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=e753246ca35494ac86549872227df731) In the illustration above, an EDIFACT connector operating in XML to EDI mode generates the document to be exchanged **(1)** and holds it in a Pending ACK state while the document is transmitted to the trading partner. The trading partner processes the transmission according to their business logic and creates acknowledgments in accordance with the configured exchange parameters **(2)**. When the acknowledgments are returned, they are processed according to the information in the next section **(3)**. #### Processing Acknowledgments In a typical flow, EDIFACT ACKs arrive at an EDIFACT connector operating in EDI to XML mode. This EDIFACT connector can be configured to automatically route any received acknowledgments to the EDIFACT connector that originally generated the document being acknowledged. Routing ACKs between EDIFACT connectors is configured visually on the Flows canvas by dragging the gray dot at the bottom of the EDIFACT connector in EDI to XML mode onto the EDIFACT connector that is in XML to EDI mode. Once the EDIFACT connector in XML to EDI mode receives the routed ACK, it pairs the ACK to the original message and changes its state from Pending ACK to Sent. #### Generating Acknowledgements When an EDIFACT connector in EDI to XML mode receives a message and generates the corresponding XML, it can automatically generate CONTRL acknowledgments for the received message. To accomplish this, check **Technical acknowledgment (CONTRL)** and/or **Functional acknowledgment (CONTRL)** in the [Acknowledgments](#acknowledgments) section of the Settings tab. These acknowledgments must be routed to another EDIFACT connector (in XML to EDI mode) to finalize the ACK. The connector to which the ACK is routed applies interchange headings and passes the ACK along to the next connector in the flow like any other EDIFACT message. To route the ACK appropriately, drag the gray dot at the bottom of the EDIFACT connector in EDI to XML mode onto the EDIFACT connector that is in XML to EDI mode. The following diagram shows the logical flow involved when creating an ACK for a received invoice document: ![EDIFACT Create ACKs](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-create.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=3d2adfeb1f049885a0cf400ccc9b6cb5) As shown in the illustration above, after a trading partner sends a message expecting an acknowledgment **(1)**, the EDIFACT connector in EDI to XML mode that parses the document automatically generates an ACK **(2)**. This ACK is an XML file containing the transactional information relevant for the transaction. Then, this XML acknowledgment is routed back to an EDIFACT connector in XML to EDI mode **(3)** so that the interchange headers (and any other EDI party agreement settings) are applied before the ACK is sent back to the trading partner. ## Master-Detail Hierarchy: Translating CPS and HYN Loops In EDI documents, most hierarchical relationships are represented by the order of EDI segments. Some EDI structures, like CPS loops (found in DESADV documents) and HYN loops (found in PRICAT documents), do not follow this convention. Instead, they have the hierarchical relationships embedded in the EDI element data itself. This can make it difficult to preserve these hierarchical relationships when converting the EDI data to XML. The EDIFACT connector supports preserving hierarchical relationships in CPS and HYN segments via the **Nest Loops** setting on the [Advanced tab](#advanced-tab). When enabled, the connector parses the elements in the CPS and HYN segments to determine which segments "belong to" other segments in a hierarchical relationship. These hierarchical relationships are reflected in the output XML as parent-child relationships; in other words, this setting converts hierarchy implied by the EDI content into hierarchy represented by the XML structure. This section explains how hierarchy is encoded in CPS and HYN data, and how this hierarchy is converted into XML when **Nest Loops** is checked. ### CPS and HYN Hierarchies All CPS and HYN segments have two values that help establish hierarchy: * an Id value, which identifies the current segment (this value is stored in CPS01 or HYN01, or as the first element in a CPS or HYN segment) * a parent-Id value, which identifies the current segment's hierarchical parent (this value is stored in CPS02 or HYN02, or as the second element in a CPS or HYN segment) For example, a segment has an Id value of '2'. If the next segment should "belong to" this prior segment in a hierarchical relationship, the next segment's parent-Id should also be '2'. If a segment's parent-Id is '0', the segment does not have a parent (it is at the top level of the hierarchy). ### Converting CPS and HYN Hierarchies to XML When **Nest Loops** is checked, the EDIFACT connector automatically handles the conversion of CPS and HYN hierarchies to the XML hierarchy. The connector parses the Id and parent-Id values from the CPS and HYN segments, and ensures that the resulting XML elements are appropriately nested inside the elements representing their parents. In other words, if `segmentA` has a parent-Id value that is equal to `segmentB`, the resulting XML has `segmentA` as a child of `segmentB`. In this way, the hierarchical relationships are preserved in the XML structure when the EDI data is translated. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## EDIFACT Operations In addition to the [Operations](../scripting/operations/operations) provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except that they must be called through the `connector.rsc` endpoint and must include an auth token. The following operation is specific to the functionality of the EDIFACT connector. ### edifactScan Scans header values from the headers of an EDIFACT document. #### Required Parameters * **file**: The path to the EDIFACT file. #### Optional Parameters * **format**: The file format. The default value is EDIFACT. #### Output Attributes * **InterchangeSyntaxIdentifier**: The Id of the syntax version being used (UNB1.1). * **InterchangeSyntaxVersion**: The version of the syntax rules being used. Values range from 1 to 5, where higher numbers support more advanced character sets (UNB1.2). * **InterchangeSenderIdQualifier**: The interchange qualifier of the sender Id (UNB2.2). * **InterchangeSenderId**: The interchange sender Id (UNB2.1). * **InterchangeReceiverIdQualifier**: The interchange qualifier of the receiver Id (UNB3.2). * **InterchangeReceiverId**: The interchange receiver Id (UNB3.1). * **DocumentType**: The document type or transaction set Identifier code (UNH2.1). ## Common Errors Following is a list of common errors, their causes, and the recommended solution. ### Error: Schema file not found (\[schema major version] \[schema document type]) **Cause** When Arc parses an EDIFACT file, it first scans the file for the EDI version and document type to determine which schema to use during parsing. The application includes an extensive set of EDIFACT schemas in JSON format. They are stored in the `schemas` folder in the application directory (next to the `data` folder). This error indicates that the application could not find a schema file matching the major version and document type reported by the file. If the \[schema major version] or \[schema document type] values are missing in the error message, the EDI document does not contain the major version or document type. Otherwise, this error indicates that the appropriate document schema is not included in the set of schema definitions automatically included with Arc. **Resolution** If the EDI file does not contain the schema major version or document type, contact the trading partner (or other source of the EDI document) and inform them that these values must be included in the message header. Otherwise, find the appropriate schema version from our free download page. Download the schema folder and place it in the following path: `[application_directory]\schemas\edifact_schemas\[major_version]\` ### Error: The segment at line \[line number] with tag \[segment name] is not in a valid position in the specified schema (\[schema major version] \[schema document type]). This may indicate that the input file contains segments that are out of order. **Cause** When Arc parses an EDIFACT file, it first scans the file for the EDI version and document type to determine which schema to use during parsing. The application includes an extensive set of EDIFACT schemas in JSON format, stored in the `www\app_data` folder of the root installation directory. This error indicates that the parser encountered an EDI segment in the file that does not match the order of segments defined in the relevant document schema. This can indicate a few different issues: * The EDI file was generated incorrectly by the trading partner * The EDI file was generated using a different schema version than what is indicated in the file * The EDI file was generated using a custom schema **Resolution** If the EDI file was generated using a custom schema, place the custom schema file (in JSON format) in the folder where Arc reads EDI schemas. The connector searches for schemas by checking the following locations in this order: 1. Inside its folder on disk (for example, `/Resources/Schemas/`) 2. If a schema is used by multiple connectors, CData recommends you use a folder structure like this: `/schemas/_schemas/`. In this example, `` should be `edifact`. If the EDI file was not generated using a custom schema, there are two approaches to resolving the discrepancy between the EDI file generated by the partner and the EDI schema used to parse it. 1. Manually edit the appropriate EDI schema file (found in the filepath above) to adjust the segment order so that it matches the partner's EDI files. The error message indicates the line in the file where the segments diverge from the schema definition. Trace through the JSON definition of segment order until the divergence is found, and re-order the segments in the JSON to match the file provided by the partner. 2. Contact the trading partner about the EDI file. Confirm that the file was generated according to the major version that is reported by the file, and explain that the order of the segments in the file is not recognized. Which approach to use depends on your familiarity with EDI segments for the particular document type (to correctly edit the JSON schema according to the partner's file) and the partner's flexibility (to adjust the EDI file in response to informing them about the error). # Email Receive Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/email-receive Retrieves messages from an IMAP mail server with OAuth 2.0 support, advanced filtering, and configurable message retention and deletion policies. The Email Receive connector supports retrieving messages from an IMAP mail server. ## Key Capabilities * IMAP-based email polling with OAuth 2.0 support for modern providers * Advanced filtering with IMAP RFC compliant search filters * Support for downloading full messages or attachments only * Configurable message retention and deletion policies ## Overview The Email Receive connector provides a simple way to automate the process of receiving emails. The connector connects to an IMAP server and polls a specified mailbox for new messages to process. By default the connector only receives messages that have not yet been processed, and it supports additional IMAP filters. The connector supports receiving entire messages in EML format or only downloading attachments on the emails being processed. > [!NOTE] > > OAuth 2.0 configuration for Outlook and Gmail mail servers are the two most common connection configurations made in the Email Receive connector. You can access additional guides and resources for these two mail servers in the CData Community. See [Using OAuth 2.0 in the Email Send and Receive Connectors](https://community.cdata.com/cdata-arc-48/using-oauth-2-0-in-the-email-send-and-receive-connectors-106?tid=106\&fid=48). ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Email Receive Configuration *Settings related to establishing the connection.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Establish a Connection To allow Arc to receive emails, you must first establish an Email Receive connection. Once established, connections can be shared across multiple Email Receive connectors. There are several ways to establish this connection: 1. Add an Email Receive connector to your flow. Then use the **Connection** dropdown to select an existing connection or click **Create** to add a new connection. 2. Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Email Receive**, and click **Next**. > [!NOTE] > > If you create the connection this way, you must still select it from the **Connection** dropdown on the connector. Complete the following properties: | Setting | Description | | - | - | | **Host** | The hostname or IP address of the IMAP server. | | **Port** | The port on which to connect to the IMAP server. Typical values include 143 and 993. | | **TLS Type** | The method used to negotiate SSL/TLS when connecting to the server: **STARTTLS**: A plain text connection is established. Afterwards, SSL/TLS can be started with an explicit command. **SSL/TLS**: Negotiation occurs immediately without first establishing a plain text connection. **None (Plain Text)**: A plain text connection is established, and no SSL/TLS is used. | | **User** | The username to login to the IMAP server. | | **Authentication Mechanism** | The type of authentication to use to login to the server. The options vary based on your choice. | | **Password** | (Basic and NTLM) The password to the IMAP server. | | **Auth URL** | (OAuth 2.0) The authorization URL for the service. This is the URL you are directed to when clicking the **Connect** button to log in and grant permission to the application. | | **Access Token URL** | (OAuth 2.0) The URL to the access token. | | **Client Id** | (OAuth 2.0) The client Id assigned when you registered for OAuth. | | **Client Secret** | (OAuth 2.0) The secret assigned when you registered for OAuth. | | **Scope** | (OAuth 2.0) A list of scopes that the application should have access to. | | **Callback URL** | (OAuth 2.0) The URL where the service you connect to returns tokens to provide access. Services often require you to whitelist the URLs that are allowed, so this is the value to use when configuring the application registration with your service. | The following settings are required to establish a network-level connection to the IMAP server: * Host * Port * TLS Type Once network-level connectivity has been established, the following settings are required to access the messages stored on the IMAP server: * User * Password or OAuth 2.0 authentication credentials * Mailbox Use the **Add and Test** button to verify that the connector can reach the server. #### Download *Settings related to processing messages.* | Setting | Description | | - | - | | **Mailbox** | The IMAP mailbox to poll for messages. | | **Only process new messages** | Check this to only process new messages. This is usually controlled by tracking the UID of downloaded messages. If the UID values become invalid, the connector falls back to using the timestamp of the last processed message or the **Days to Search** setting if no messages have been processed. | | **Days to Search** | The number of days to search for messages. If **Only process new messages** is checked, this value is only used if the connector cannot search by UID or the timestamp of the last processed message. | | **Additional Search Filters** | Supply an IMAP RFC compliant search filter to append to the query that is automatically built based on the **New Messages** and **Days to Search** settings. | | **Download Type** | Choose to download full messages in EML format or only email attachments. | | **Delete Emails** | Check this to delete successfully downloaded messages from the mailbox. | | **Move to Folder** | The mailbox folder to move messages to after they are successfully downloaded. If the specified folder does not exist, it is created automatically. If set, **Delete Emails** has no effect. You can use [Macros](#macros) in the folder path. | | **Expunge After Delete** | Check this to permanently delete successfully downloaded emails from the mailbox. **Warning**: The server might permanently delete (expunge) messages immediately, making them unrecoverable. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **IMAP Headers** | A comma-delimited list of headers from the IMAP message that should be promoted as metadata in the downloaded file. | | **Incoming Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. | | **Command Timeout (seconds)** | The command execution timeout duration in seconds. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Receive** | Whether the connector should automatically poll the remote mailbox for messages to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by tracked headers, time, and/or status. ## Receiving Emails If **Receive Automation** is enabled, the Email Receive connector automatically polls the configured IMAP mailbox for messages to download. By default, the connector only receives messages it has not already downloaded. The first time the connector receives messages, use the **Days to Search** setting in the [Download](#download) configuration section to specify the range of messages to download. Messages can also be manually retrieved by clicking the **Receive** button on the **Transactions** tab of the connector. The **Download Type** setting determines whether full messages or just message attachments are downloaded. Downloaded files are placed in the **Output Folder** or passed along to the next connector in the flow. When **Download Type** is set to *Attachments*, the connector adds two headers to each message: * EmailReceive-Attachment-Index: The attachment number for each attachment * EmailReceive-Attachment-Count: The total number of attachments on the email ### Search Filters The **IMAP Search Filter** setting allows for further customization of the range of messages to retrieve. This field accepts standard IMAP protocol filters and only downloads messages that match the configured filters. The following filters are supported: | Filter | Description | | - | - | | `` | Messages with message sequence numbers corresponding to the specified message sequence number set. | | **ALL** | All messages in the mailbox: this is the default initial key for AND-ing. | | **ANSWERED** | Messages with the `\Answered` flag set. | | **BCC** `` | Messages that contain the specified string in the envelope structure's BCC field. | | **BEFORE** `` | Messages whose internal date is earlier than the specified date. | | **BODY** `` | Messages that contain the specified string in the body of the message. | | **CC** `` | Messages that contain the specified string in the envelope structure's CC field. | | **DELETED** | Messages with the `\Deleted` flag set. | | **DRAFT** | Messages with the `\Draft` flag set. | | **FLAGGED** | Messages with the `\Flagged` flag set. | | **FROM** `` | Messages that contain the specified string in the envelope structure's FROM field. | | **HEADER** ` ` | Messages that have a header with the specified field-name (as defined in \[RFC-822]) and that contain the specified string in the \[RFC-822] field-body. | | **KEYWORD** `` | Messages with the specified keyword set. | | **LARGER** `` | Messages with an RFC822.SIZE larger than the specified number of octets. | | **NEW** | Messages that have the `\Recent` flag set but not the `\Seen` flag. This is functionally equivalent to "(RECENT UNSEEN)". | | **NOT** `` | Messages that do not match the specified search key. | | **OLD** | Messages that do not have the `\Recent` flag set. This is functionally equivalent to "NOT RECENT" (as opposed to "NOT NEW"). | | **ON** `` | Messages whose internal date is within the specified date. | | **OR** ` ` | Messages that match either search key. | | **RECENT** | Messages that have the `\Recent` flag set. | | **SEEN** | Messages that have the `\Seen` flag set. | | **SENTBEFORE** `` | Messages whose \[RFC-822] Date: header is earlier than the specified date. | | **SENTON** `` | Messages whose \[RFC-822] Date: header is within the specified date. | | **SENTSINCE** `` | Messages whose \[RFC-822] Date: header is within or later than the specified date. | | **SINCE** `` | Messages whose internal date is within or later than the specified date. | | **SMALLER** `` | Messages with an RFC822.SIZE smaller than the specified number of octets. | | **SUBJECT** `` | Messages that contain the specified string in the envelope structure's SUBJECT field. | | **TEXT** `` | Messages that contain the specified string in the header or body of the message. | | **TO** `` | Messages that contain the specified string in the envelope structure's TO field. | | **UID** `` | Messages with unique identifiers corresponding to the specified unique identifier set. | | **UNANSWERED** | Messages that do not have the `\Answered` flag set. | | **UNDELETED** | Messages that do not have the `\Deleted` flag set. | | **UNDRAFT** | Messages that do not have the `\Draft` flag set. | | **UNFLAGGED** | Messages that do not have the `\Flagged` flag set. | | **UNKEYWORD** `` | Messages that do not have the specified keyword set. | | **UNSEEN** | Messages that do not have the `\Seen` flag set. | ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Email Receive Connector (Deprecated) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/email-receivev1 The Email Receive connector supports retrieving messages from an IMAP mail server with OAuth 2.0 support and advanced filtering. The Email Receive connector supports retrieving messages from an IMAP mail server. ## Key Capabilities * IMAP-based email polling with OAuth 2.0 support for modern providers * Advanced filtering with IMAP RFC compliant search filters * Support for downloading full messages or attachments only * Configurable message retention and deletion policies ## Overview The Email Receive connector provides a simple way to automate the process of receiving emails. The connector connects to an IMAP server and polls a specified mailbox for new messages to process. By default the connector only receives messages that have not yet been processed, and it supports additional IMAP filters. The connector supports receiving entire messages in EML format or only downloading attachments on the emails being processed. > [!NOTE] > > OAuth 2.0 configuration for Outlook and Gmail mail servers are the two most common connection configurations made in the Email Receive connector. You can access additional guides and resources for these two mail servers in the CData Community. See [Using OAuth 2.0 in the Email Send and Receive Connectors](https://community.cdata.com/cdata-arc-48/using-oauth-2-0-in-the-email-send-and-receive-connectors-106?tid=106\&fid=48). ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Email Receive Configuration *Settings related to establishing the connection.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### IMAP Connection *Settings related to establishing the connection to the IMAP server.* | Setting | Description | | - | - | | **Host** | The hostname or IP address of the IMAP server. | | **Port** | The port on which to connect to the IMAP server. Typical values include 143 and 993. | | **Mailbox** | The IMAP mailbox to poll for messages. | | **TLS Type** | The method used to negotiate SSL/TLS when connecting to the server: **STARTTLS**: A plain text connection is established. Afterwards, SSL/TLS can be started with an explicit command. **SSL/TLS**: Negotiation occurs immediately without first establishing a plain text connection. **None (Plain Text)**: A plain text connection is established, and no SSL/TLS is used. | ### Authentication *Settings related to authenticating the connection.* | Setting | Description | | - | - | | **User** | The username to login to the IMAP server. | | **Authentication Mechanism** | The type of authentication to use to login to the server. The options vary based on your choice. | | **Password** | (Basic and NTLM) The password to the IMAP server. | | **Auth URL** | (OAuth 2.0) The authorization URL for the service. This is the URL you are directed to when clicking the **Connect** button to log in and grant permission to the application. | | **Access Token URL** | (OAuth 2.0) The URL to the access token. | | **Client Id** | (OAuth 2.0) The client Id assigned when you registered for OAuth. | | **Client Secret** | (OAuth 2.0) The secret assigned when you registered for OAuth. | | **Scope** | (OAuth 2.0) A list of scopes that the application should have access to. | | **Callback URL** | (OAuth 2.0) The URL where the service you connect to returns tokens to provide access. Services often require you to whitelist the URLs that are allowed, so this is the value to use when configuring the application registration with your service. | #### Download *Settings related to processing messages.* | Setting | Description | | - | - | | **New Messages** | Check this to only process new messages. This is usually controlled by tracking the UID of downloaded messages. If the UID values become invalid, the connector falls back to using the timestamp of the last processed message or the **Days to Search** setting if no messages have been processed. | | **Days to Search** | The number of days to search for messages. If **Only process new messages** is checked, this value is only used if the connector cannot search by UID or the timestamp of the last processed message. | | **Additional Search Filters** | Supply an IMAP RFC compliant search filter to append to the query that is automatically built based on the **New Messages** and **Days to Search** settings. | | **Download Type** | Choose to download full messages in EML format or only email attachments. | | **Delete Emails** | Check this to delete successfully downloaded messages from the mailbox. | | **Expunge After Delete** | Check this to permanently delete successfully downloaded emails from the mailbox. **Warning**: The server might permanently delete (expunge) messages immediately, making them unrecoverable. | ### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Receive** | Whether the connector should automatically poll the remote mailbox for messages to download. | | **Receive Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Advanced Tab #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **IMAP Headers** | A comma-delimited list of headers from the IMAP message that should be promoted as metadata in the downloaded file. | | **Incoming Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. | | **Command Timeout (seconds)** | The command execution timeout duration in seconds. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Establishing a Connection The following settings are required to establish a network-level connection to the IMAP server: * Host * Port * TLS Type Use the **Test Connection** button to verify that the connector can reach the server. Once network-level connectivity has been established, the following settings are required to access the messages stored on the IMAP server: * User * Password or OAuth 2.0 authentication credentials * Mailbox ## Receiving Emails If **Receive Automation** is enabled, the Email Receive connector automatically polls the configured IMAP mailbox for messages to download. By default, the connector only receives messages it has not already downloaded. The first time the connector receives messages, use the **Days to Search** setting in the [Download](#download) configuration section to specify the range of messages to download. Messages can also be manually retrieved by clicking the **Receive** button on the Output tab of the connector. The **Download Type** setting determines whether full messages or just message attachments are downloaded. Downloaded files are placed in the **Output Folder** or passed along to the next connector in the flow. When **Download Type** is set to *Attachments*, the connector adds two headers to each message: * EmailReceive-Attachment-Index: The attachment number for each attachment * EmailReceive-Attachment-Count: The total number of attachments on the email ### Search Filters The **IMAP Search Filter** setting allows for further customization of the range of messages to retrieve. This field accepts standard IMAP protocol filters and only downloads messages that match the configured filters. The following filters are supported: | Filter | Description | | - | - | | `` | Messages with message sequence numbers corresponding to the specified message sequence number set. | | **ALL** | All messages in the mailbox: this is the default initial key for AND-ing. | | **ANSWERED** | Messages with the `\Answered` flag set. | | **BCC** `` | Messages that contain the specified string in the envelope structure's BCC field. | | **BEFORE** `` | Messages whose internal date is earlier than the specified date. | | **BODY** `` | Messages that contain the specified string in the body of the message. | | **CC** `` | Messages that contain the specified string in the envelope structure's CC field. | | **DELETED** | Messages with the `\Deleted` flag set. | | **DRAFT** | Messages with the `\Draft` flag set. | | **FLAGGED** | Messages with the `\Flagged` flag set. | | **FROM** `` | Messages that contain the specified string in the envelope structure's FROM field. | | **HEADER** ` ` | Messages that have a header with the specified field-name (as defined in \[RFC-822]) and that contain the specified string in the \[RFC-822] field-body. | | **KEYWORD** `` | Messages with the specified keyword set. | | **LARGER** `` | Messages with an RFC822.SIZE larger than the specified number of octets. | | **NEW** | Messages that have the `\Recent` flag set but not the `\Seen` flag. This is functionally equivalent to "(RECENT UNSEEN)". | | **NOT** `` | Messages that do not match the specified search key. | | **OLD** | Messages that do not have the `\Recent` flag set. This is functionally equivalent to "NOT RECENT" (as opposed to "NOT NEW"). | | **ON** `` | Messages whose internal date is within the specified date. | | **OR** ` ` | Messages that match either search key. | | **RECENT** | Messages that have the `\Recent` flag set. | | **SEEN** | Messages that have the `\Seen` flag set. | | **SENTBEFORE** `` | Messages whose \[RFC-822] Date: header is earlier than the specified date. | | **SENTON** `` | Messages whose \[RFC-822] Date: header is within the specified date. | | **SENTSINCE** `` | Messages whose \[RFC-822] Date: header is within or later than the specified date. | | **SINCE** `` | Messages whose internal date is within or later than the specified date. | | **SMALLER** `` | Messages with an RFC822.SIZE smaller than the specified number of octets. | | **SUBJECT** `` | Messages that contain the specified string in the envelope structure's SUBJECT field. | | **TEXT** `` | Messages that contain the specified string in the header or body of the message. | | **TO** `` | Messages that contain the specified string in the envelope structure's TO field. | | **UID** `` | Messages with unique identifiers corresponding to the specified unique identifier set. | | **UNANSWERED** | Messages that do not have the `\Answered` flag set. | | **UNDELETED** | Messages that do not have the `\Deleted` flag set. | | **UNDRAFT** | Messages that do not have the `\Draft` flag set. | | **UNFLAGGED** | Messages that do not have the `\Flagged` flag set. | | **UNKEYWORD** `` | Messages that do not have the specified keyword set. | | **UNSEEN** | Messages that do not have the `\Seen` flag set. | ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Email Send Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/email-send Sends messages via an SMTP mail server or SendGrid with OAuth 2.0 support, template-driven content generation, and attachment mode for file delivery. The Email Send connector sends messages via an SMTP mail server. ## Key Capabilities * SMTP-based email delivery using a user-specified server or SendGrid, with OAuth 2.0 support for modern providers * Template mode for dynamic content generation from XML data or ArcScript * Attachment mode for sending files with customizable message bodies * Support for multiple recipients (To, CC, and BCC) with dynamic field population ## Overview An Email Send connector connects to an SMTP server (user-specified or SendGrid) delivery service to send files as outgoing email messages. Input files processed by the connector can be sent directly as email attachments, or data can be parsed from input files and included in the outgoing message via a scripting template. For more information on creating a scripting template, see the [Templates](#template-mode) sections. > [!NOTE] > > OAuth 2.0 configuration for Outlook and Gmail mail servers are the two most common connection configurations made in the Email Send connector. You can access additional guides and resources for these two mail servers in the CData Community. See [Using OAuth 2.0 in the Email Send and Receive Connectors](https://community.cdata.com/cdata-arc-48/using-oauth-2-0-in-the-email-send-and-receive-connectors-106?tid=106\&fid=48). ## Connector Configuration This section contains all of the configurable connector properties. ### Email Config #### Establish a Connection To allow Arc to send emails, you must first establish an Email Send connection. Once established, connections can be shared across multiple Email Send connectors. There are two ways to establish this connection: 1. Add an Email Send connector to your flow. Then use the **Connection** dropdown to select an existing connection or click **Create** to add a new connection. 2. Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Email Send**, and click **Next**. > [!NOTE] > > If you create the connection this way, you must still select it from the **Connection** dropdown on the connector. Complete the following properties: | Setting | Description | | - | - | | **Email Protocol** | Whether to use a user-specified SMTP server or SendGrid. If you choose SendGrid, only **TLS Type** remains. All other fields are SMTP-related. | | **API Key** | Your SendGrid API key. Not applicable for Arc Cloud implementations. | | **Host** | The hostname or IP address of the SMTP server. | | **Port** | The port on which to connect to the SMTP server. Typical values include 143 and 993. | | **Authentication Mechanism** | The type of authentication to use to login to the server. The options vary based on your choice. | | **Password** | (Basic and NTLM) The password to the IMAP server. | | **Auth URL** | (OAuth 2.0) The authorization URL for the service. This is the URL you are directed to when clicking the **Connect** button to log in and grant permission to the application. | | **Access Token URL** | (OAuth 2.0) The URL to the access token. | | **Client Id** | (OAuth 2.0) The client Id assigned when you registered for OAuth. | | **Client Secret** | (OAuth 2.0) The secret assigned when you registered for OAuth. | | **Scope** | (OAuth 2.0) A list of scopes that the application should have access to. | | **Callback URL** | (OAuth 2.0) The URL where the service you connect to returns tokens to provide access. Services often require you to whitelist the URLs that are allowed, so this is the value to use when configuring the application registration with your service. | | **None** | No authentication is required. | | **User** | The username to login to the SMTP server. Not visible when the **Authentication Mechanism** is set to None. | | **TLS Type** | The method used to negotiate TLS/SSL when connecting to the server: **STARTTLS**: A plain text connection is established. Afterwards, TLS/SSL can be started with an explicit command. **SSL/TLS**: Negotiation occurs immediately without first establishing a plain text connection. **None (Plain Text)**: A plain text connection is established, and no TLS/SSL is used. **Automatic**: The appropriate TLS/SSL negotiation method is detected and applied based on the port number. STARTTLS is used for standard mail ports (25, 587), and TLS/SSL is used for port 465. | The following settings are required to establish a network-level connection to the SMTP server: * Host * Port * TLS Type Once network-level connectivity has been established, the following settings are required to access the messages stored on the server: * User * Password or OAuth 2.0 authentication credentials * Mailbox The only setting SendGrid expects is TLS Type. Use the **Add and Test** button to verify that the connector can reach the server. #### Additional Configuration *Settings for the outgoing email message. Most of these fields can include dynamic data from the outgoing file (see [Dynamic Fields](#dynamic-fields) for more information).* | Setting | Description | | - | - | | **Message Type** | Specifies the method for handling the message: **Attachment**: Send files that arrive at the connector as attachments. See [Attachment Mode](#attachment-mode) for more information and an example. **Template**: Interpret the message body as an ArcScript template that reads values from the input file. For more information on creating a scripting template, see [Template Mode](#template-mode). | | **To** | The email address of the outgoing email message recipient. Use a comma-separated list to send to multiple recipients. | | **Show From** | Opens a field to enter the email address of the sender of the outgoing email message. The connector attempts to automatically determine the value for this field based on the **User** credential passed to the SMTP server. | | **Show Cc / Show Bcc** | Opens fields to enter an email address to carbon copy (CC) or blind carbon copy (BCC). Use a comma-separated list to send to multiple recipients. | | **Subject** | The subject line for the outgoing email message. | | **Message Body** | The email body. | | **Allow ArcScript in Email Body** | Check this to permit the evaluation of ArcScript in the message body when the **Message Type** is Attachment. | > [!NOTE] > > You can also use ArcScript in the **To**, **From**, **CC**, **BCC** and **Subject** fields by clicking the ArcScript editor button. ### Settings Tab #### Email Send Configuration *Settings for the name and description of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Advanced Settings *Extra settings related to processing.* | Setting | Description | | - | - | | **Attachment Filter** | A glob pattern filter to apply to messages before they are sent. The pattern can be positive or negative. For example, `*.edi` only sends files with a `.edi` extension and `-*.tmp` excludes files with a `.tmp` extension. You can provide a comma-separated list of filters. Later filters take priority unless you use an exact match. | | **Outgoing Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. | | **Extension Map** | A comma separated list of name-value pairs for determining the Content-Type header value for attachments. For example, *.txt=text/plain, .dat=application/edi-x12, .edi=application/edifact* **Note**: Including a Content-Type header on input messages overrides the content type determined by this setting. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by tracked headers, time, and/or status. ## Sending Emails The [Email Config](#email-config) tab determines the email message to be sent: * If **Message Type** is set to **Attachment**, files processed by the connector are sent as email attachments, and you should specify the body of the message in the message text editor. See [Attachment Mode](#attachment-mode) for more information. * If **Message Type** is set to **Template**, the message text editor is treated as an ArcScript template. In the scripting template, values can be parsed out from the files processed by the connector and included in the email body. See [Template Mode](#template-mode) for more information. To send multiple attachments in a single email, you need to construct a [batch group](../getting-started/architecture#batch-groups) containing all the attachments you want to include, then send that batch group to the Email Send connector for processing. ### Dynamic Fields The **To**, **From**, **CC**, **BCC**, and **Subject** fields support dynamic information from the outgoing file (for example, the filename and the connectors that have processed the file). The file information is available through the [*\_message*](../scripting/introduction-to-arcscript#message-_message) item. The *\_message* item has a collection of attributes with a `header:` prefix that corresponds to each of the headers available in the outgoing file. These headers include the filename, any metadata related to processing the file within the Arc flow, and any custom headers promoted to the file via scripting. The *\_message* item and the header attributes are referenced in square brackets like any other ArcScript item: `[_message.header:headerName]` For example, the Subject could be set to the following value to include the filename of the outgoing file: `[_message.header:filename]` ### Template Mode Setting **Message Type** to Template instructs the connector to interpret the message body as an ArcScript template. The template uses a simple syntax to interact with the input file and retrieve values to use in the email body. These values can be combined with static text to generate a standard body format that is dynamically populated based on the files processed by the connector. To use Template mode, the connector must receive an XML file as an input. Templates can only parse information from XML files. #### Basic Template For an example of a template message, consider the following input XML document: ```xml theme={null} ABC, Inc. john@abcinc.com John Blue Widget ``` This file contains a list of attributes—`Company`, `Email`, `FirstName`, `Description`—inside of `Lead` tags. The ArcScript expression to pass the `Email` field into a message would look like this: `[document.lead_email]` There are three components to this expression: * The *document* item, which points to the currently open file. * The *lead\_email* attribute, which points to '[john@abcinc.com](mailto:john@abcinc.com)'. * Square brackets (`[]`), which indicate the use of a formatter rather than literal characters. An email body template for the XML input file above might look like this: ``` Hi [document.lead_firstname],
We saw that you might be interested in the following product:
[document.lead_description]
Would you like to set up a demo to learn more about how we can help you with your work at [document.lead_company]? ``` This message would generate the following email: ``` Hi John, We saw that you might be interested in the following product: Blue Widget Would you like to set up a demo to learn more about how we can help you with your work at ABC, Inc.? ``` Refer to the [Scripting](../scripting/scripting) section for more information about ArcScript. #### XPath Formatting The XPath formatter (written as `xpath` in expressions) provides an alternative approach to retrieving dynamic values from files. This approach might be more familiar to users already using ArcScript elsewhere in Arc. XPath requires input files to be XML files. Using the same input file example as above: ```xml theme={null} ABC, Inc. john@abcinc.com John Blue Widget ``` The XPath expression to return the email address would be: `[xpath(/Lead/Email)]` There are three components to this expression: * *xpath* followed by parentheses, which denotes the expression as an XPath expression. * The path to the data, with each level delineated by slashes. * Square brackets (`[]`), which indicate the use of a formatter rather than literal characters. ##### Using the XPath Formatter in Template Mode Consider the following XML: ```xml theme={null} myProject Bob Ross 02/27/2024 http://mydomain.com/desc http://mydomain.com/images http://mydomain.com/colors ``` An email body template that uses xpath expressions for the XML input file above might look like this: ```xml theme={null} Hi [xpath(/Items/assignee)],
You have a new project to start working on. The name of the project is [xpath(/Items/projectName)].
All the resources you need can be found at the following links:
  • Description: [xpath(/Items/resources/description)]
  • Images: [xpath(/Items/resources/images)]
  • Colors: [xpath(/Items/resources/colors)]

The due date for this project is [xpath(/Items/dueDate)]. ``` This message would generate the following email: ``` Hi Bob Ross, You have a new project to start working on. The name of the project is myProject. All the resources you need can be found at the following links: - Description: http://mydomain.com/desc - Images: http://mydomain.com/images - Colors: http://mydomain.com/colors The due date for this project is 02/27/2024. ``` #### Code View Editor The Code View editor enables the creation of custom ArcScript for advanced processing of information from XML files. To open the Code View editor, click the **Code View** button in the message editor. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/code_view_editor.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=f4456322190263f680264dcc2c32a89d) In this mode, ArcScript expressions can be used to process multiple data entries at once. The following input file example includes a list of names: ```xml theme={null} Dave James Satchel John Smith Gina Newberry ``` The following message written in the Code View editor incorporates data from this input file using ArcScript: ```xml theme={null} Hi [document.record_to],
Here are the people you were asking about:
[_value] [_value]
``` This code loops over all of the first and last name values in the input XML file and prints them in the body of the message. When run with the input file above, the final email looks like this: ``` Hi Dave, Here are the people you were asking about: James Satchel John Smith Gina Newberry ``` Refer to the [Scripting](../scripting/scripting) section for more information about ArcScript and custom expressions. ### Attachment Mode If you set **Message Type** to Attachment, type your email message body into the editor. If necessary, you can reference headers on the input message to enrich the body of the email. The Email Send connector then evaluates those headers, and attaches the input file to the outbound email. > [!NOTE] > > This relies on a previous step in the flow to promote the necessary values as headers onto the message. Depending on the content of the file, you can do this with various [Operations](../scripting/operations/operations) and connector settings. Consider the following XML: ```xml theme={null} myProject Bob Ross 02/27/2024 http://mydomain.com/desc http://mydomain.com/images http://mydomain.com/colors ``` The email body might look like this: ```xml theme={null} Hi [_message.header:assignee],
You have a new project to start working on. The name of the project is [_message.header:projectname].
All the resources you need can be found at the following links:
  • Description: [_message.header:description]
  • Images: [_message.header:images]
  • Colors: [_message.header:colors]

The due date for this project is [_message.header:duedate]. ``` This message would generate the following email, and include the input file as an attachment: ``` Hi Bob Ross, You have a new project to start working on. The name of the project is myProject. All the resources you need can be found at the following links: - Description: http://mydomain.com/desc - Images: http://mydomain.com/images - Colors: http://mydomain.com/colors The due date for this project is 02/27/2024. ``` # Email Send Connector (Deprecated) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/email-sendv1 The Email Send connector sends messages via an SMTP mail server with OAuth 2.0 support and template-based dynamic content. The Email Send connector sends messages via an SMTP mail server. ## Key Capabilities * SMTP-based email delivery with OAuth 2.0 support for modern providers * Template mode for dynamic content generation from XML data or ArcScript * Attachment mode for sending files with customizable message bodies * Support for multiple recipients (To, CC, and BCC) with dynamic field population ## Overview An Email Send connector connects to an SMTP server to send files as outgoing email messages. Input files processed by the connector can be sent directly as email attachments, or data can be parsed from input files and included in the outgoing message via a scripting template. For more information on creating a scripting template, see the [Templates](#template-mode) sections. > [!NOTE] > > OAuth 2.0 configuration for Outlook and Gmail mail servers are the two most common connection configurations made in the Email Send connector. You can access additional guides and resources for these two mail servers in the CData Community. See [Using OAuth 2.0 in the Email Send and Receive Connectors](https://community.cdata.com/cdata-arc-48/using-oauth-2-0-in-the-email-send-and-receive-connectors-106?tid=106\&fid=48). ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Email Send Configuration *Settings for the name and description of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Connection *Settings for connecting to the SMTP server.* | Setting | Description | | - | - | | **Host** | The hostname or IP address of the SMTP server. | | **Port** | The port on which to connect to the SMTP server. | | **User** | The username to login to the SMTP server. | | **Authentication Mechanism** | The type of authentication to use to login to the server. The options vary based on your choice. | | **Password** | (Basic and NTLM) The password to the SMTP server. | | **Auth URL** | (OAuth 2.0) The authorization URL for the service. This is the URL you are directed to when clicking the **Connect** button to log in and grant permission to the application. | | **Access Token URL** | (OAuth 2.0) The URL to the access token. | | **Client Id** | (OAuth 2.0) The client Id assigned when you registered for OAuth. | | **Client Secret** | (OAuth 2.0) The secret assigned when you registered for OAuth. | | **Scope** | (OAuth 2.0) A list of scopes that the application should have access to. | | **Callback URL** | (OAuth 2.0) The URL where the service you connect to returns tokens to provide access. Services often require you to whitelist the URLs that are allowed, so this is the value to use when configuring the application registration with your service. | | **None** | No authentication is required. | | **TLS Type** | The method used to negotiate SSL/TLS when connecting to the server: **STARTTLS**: A plain text connection is established. Afterwards, SSL/TLS can be started with an explicit command. **SSL/TLS**: Negotiation occurs immediately without first establishing a plain text connection. **None (Plain Text)**: A plain text connection is established, and no SSL/TLS is used. **Automatic**: The connector determines the TLS type based on the configured port number. | #### Message *Settings for the outgoing email message. Most of these fields can include dynamic data from the outgoing file (see [Dynamic Fields](#dynamic-fields) for more information).* | Setting | Description | | - | - | | **To** | The email address of the outgoing email message recipient. Use a comma-separated list to send to multiple recipients. | | **Show From** | Opens a field to enter the email address of the sender of the outgoing email message. The connector attempts to automatically determine the value for this field based on the **User** credential passed to the SMTP server. | | **Add Cc / Add Bcc** | Opens fields to enter an email address to carbon copy (CC) or blind carbon copy (BCC). Use a comma-separated list to send to multiple recipients. | | **Subject** | The subject line for the outgoing email message. | | **Message Type** | Specifies the method for handling the message: **Attachment**: Send files that arrive at the connector as attachments. See [Attachment Mode](#attachment-mode) for more information and an example. **Template**: Interpret the message body as a scripting template that reads values from the input file. For more information on creating a scripting template, see [Template Mode](#template-mode). | ### Automation Tab #### Automation Settings | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Advanced Tab #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Advanced Settings *Extra settings related to processing.* | Setting | Description | | - | - | | **Allow ArcScript in Attachment mode** | Check this to permit the evaluation of ArcScript in the message body when the **Message Type** is Attachment. | | **Excluded Extensions** | A comma-delimited list of file extensions that should not be processed by the connector. | | **Outgoing Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. | | **Extension Map** | A comma separated list of name-value pairs for determining the Content-Type header value for attachments. For example, *.txt=text/plain, .dat=application/edi-x12, .edi=application/edifact*. **Note**: Including a Content-Type header on input messages overrides the content type determined by this setting. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Establishing a Connection The following settings are required to establish a network-level connection to the SMTP server: * Host * Port * TLS Type Click the **Test Connection** button to verify that the connector can reach the server. Once network-level connectivity has been established, the following settings are required to login to the server and begin sending emails: * User * Password ## Sending Emails The [Message](#message) section on the connector **Settings** page determines the email message to be sent: * If **Message Type** is set to **Attachment**, files processed by the connector are sent as email attachments, and you should specify the body of the message in the message text editor. See [Attachment Mode](#attachment-mode) for more information. * If **Message Type** is set to **Template**, the message text editor is treated as an ArcScript template. In the scripting template, values can be parsed out from the files processed by the connector and included in the email body. See [Template Mode](#template-mode) for more information. To send multiple attachments in a single email, you need to construct a batch group containing all the attachments you want to include, then send that batch group to the Email Send connector for processing. ### Dynamic Fields The **To**, **From**, **CC**, **BCC**, and **Subject** fields support dynamic information from the outgoing file (for example, the filename and the connectors that have processed the file). The file information is available through the \[*\_message*] item. The *\_message* item has a collection of attributes with a `header:` prefix that corresponds to each of the headers available in the outgoing file. These headers include the filename, any metadata related to processing the file within the Arc flow, and any custom headers promoted to the file via scripting. The *\_message* item and the header attributes are referenced in square brackets like any other ArcScript item: `[_message.header:headerName]` For example, the Subject could be set to the following value to include the filename of the outgoing file: `[_message.header:filename]` ### Template Mode Setting **Message Type** to Template instructs the connector to interpret the message body as an ArcScript template. The template uses a simple syntax to interact with the input file and retrieve values to use in the email body. These values can be combined with static text to generate a standard body format that is dynamically populated based on the files processed by the connector. To use Template mode, the connector must receive an XML file as an input. Templates can only parse information from XML files. #### Basic Template For an example of a template message, consider the following input XML document: ```xml theme={null} ABC, Inc. john@abcinc.com John Blue Widget ``` This file contains a list of attributes—`Company`, `Email`, `FirstName`, `Description`—inside of `Lead` tags. The ArcScript expression to pass the `Email` field into a message would look like this: `[document.lead_email]` There are three components to this expression: * The *document* item, which points to the currently open file. * The *lead\_email* attribute, which points to '[john@abcinc.com](mailto:john@abcinc.com)'. * Square brackets (`[]`), which indicate the use of a formatter rather than literal characters. An email body template for the XML input file above might look like this: ``` Hi [document.lead_firstname],
We saw that you might be interested in the following product:
[document.lead_description]
Would you like to set up a demo to learn more about how we can help you with your work at [document.lead_company]? ``` This message would generate the following email: ``` Hi John, We saw that you might be interested in the following product: Blue Widget Would you like to set up a demo to learn more about how we can help you with your work at ABC, Inc.? ``` #### XPath Formatting The XPath formatter (written as `xpath` in expressions) provides an alternative approach to retrieving dynamic values from files. This approach might be more familiar to users already using ArcScript elsewhere in Arc. XPath requires input files to be XML files. Using the same input file example as above: ```xml theme={null} ABC, Inc. john@abcinc.com John Blue Widget ``` The XPath expression to return the email address would be: `[xpath(/Lead/Email)]` There are three components to this expression: * *xpath* followed by parentheses, which denotes the expression as an XPath expression. * The path to the data, with each level delineated by slashes. * Square brackets (`[]`), which indicate the use of a formatter rather than literal characters. ##### Using the XPath Formatter in Template Mode Consider the following XML: ```xml theme={null} myProject Bob Ross 02/27/2024 http://mydomain.com/desc http://mydomain.com/images http://mydomain.com/colors ``` An email body template that uses xpath expressions for the XML input file above might look like this: ```xml theme={null} Hi [xpath(/Items/assignee)],
You have a new project to start working on. The name of the project is [xpath(/Items/projectName)].
All the resources you need can be found at the following links:
  • Description: [xpath(/Items/resources/description)]
  • Images: [xpath(/Items/resources/images)]
  • Colors: [xpath(/Items/resources/colors)]

The due date for this project is [xpath(/Items/dueDate)]. ``` This message would generate the following email: ``` Hi Bob Ross, You have a new project to start working on. The name of the project is myProject. All the resources you need can be found at the following links: - Description: http://mydomain.com/desc - Images: http://mydomain.com/images - Colors: http://mydomain.com/colors The due date for this project is 02/27/2024. ``` #### Code View Editor The Code View editor enables the creation of custom ArcScript for advanced processing of information from XML files. To open the Code View editor, click the **Code View** button in the message editor. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/code_view_editor.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=f4456322190263f680264dcc2c32a89d) In this mode, ArcScript expressions can be used to process multiple data entries at once. The following input file example includes a list of names: ```xml theme={null} Dave James Satchel John Smith Gina Newberry ``` The following message written in the Code View editor incorporates data from this input file using ArcScript: ```xml theme={null} Hi [document.record_to],
Here are the people you were asking about:
[_value] [_value]
``` This code loops over all of the first and last name values in the input XML file and prints them in the body of the message. When run with the input file above, the final email looks like this: ``` Hi Dave, Here are the people you were asking about: James Satchel John Smith Gina Newberry ``` ### Attachment Mode If you set **Message Type** to Attachment, type your email message body into the editor. If necessary, you can reference headers on the input message to enrich the body of the email. The Email Send connector then evaluates those headers, and attaches the input file to the outbound email. > [!NOTE] > > This relies on a previous step in the flow to promote the necessary values as headers onto the message. Depending on the content of the file, you can do this with various Operations and connector settings. Consider the following XML: ```xml theme={null} myProject Bob Ross 02/27/2024 http://mydomain.com/desc http://mydomain.com/images http://mydomain.com/colors ``` The email body might look like this: ```xml theme={null} Hi [_message.header:assignee],
You have a new project to start working on. The name of the project is [_message.header:projectname].
All the resources you need can be found at the following links:
  • Description: [_message.header:description]
  • Images: [_message.header:images]
  • Colors: [_message.header:colors]

The due date for this project is [_message.header:duedate]. ``` This message would generate the following email, and include the input file as an attachment: ``` Hi Bob Ross, You have a new project to start working on. The name of the project is myProject. All the resources you need can be found at the following links: - Description: http://mydomain.com/desc - Images: http://mydomain.com/images - Colors: http://mydomain.com/colors The due date for this project is 02/27/2024. ``` # Encoding Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/encoding Automatically detects a file's character encoding and converts it to a specified output encoding, supporting UTF-8, UTF-16, ISO-8859-1, and Asian character sets. The Encoding connector auto-detects a file's encoding and converts the file to a specified output encoding. ## Key Capabilities * Automatic encoding detection and conversion between multiple character sets * Support for UTF-8, UTF-16, ISO-8859-1, Windows-1252, and Asian character sets * Fallback handling for undetectable encodings with configurable error handling ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to establishing the connection.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Input Encoding** | The encoding of the input file (the default is **Auto-Detect**). The available encodings are `UTF-8`, `UTF-16`, `ISO-8859-1`, `Windows-1252`, `GB2312`, `Shift-JIS`, `GBK`, `Big5`, `EUC-JP`, and `TIS-620`. | | **Auto Detect Fallback** | The encoding to use if **Auto-Detect** fails to detect an encoding (the default is **Use Fallback Encoding**). The other options are **Keep Original** and **Failure**. | | **Fallback Encoding** | The encoding the connector falls back to if **Auto-Detect** fails to identify the encoding *and* **Auto Detect Fallback** is set to **Use Fallback Encoding** (the default is `UTF-8`). The other options are `UTF-16`, `ISO-8859-1`, `Windows-1252`, `GB2312`, `Shift-JIS`, `GBK`, `Big5`, `EUC-JP`, and `TIS-620`. | | **Output Encoding** | The encoding to apply to the output file (the default is `UTF-8`). The other options are `UTF-16`, `ISO-8859-1`, `Windows-1252`, `GB2312`, `Shift-JIS`, `GBK`, `Big5`, `EUC-JP`, and `TIS-620`. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Excel Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/excel The Excel connector converts data between Excel sheets (.xlsx) and XML files using Table or Template modes. The Excel connector converts data between Excel sheets (.xlsx) and XML files. ## Key Capabilities * Bidirectional Excel/XML conversion in both Table and Template modes * Advanced template system using ArcScript for dynamic data population * Support for multiple worksheets with configurable row/column ranges ## Overview Excel connectors can be configured in two modes: [Table](#table-mode) and [Template](#template-mode). These modes are described below. ### Table Mode The default mode for Excel connectors is **Table**. This mode only supports converting files from XLSX to XML. To convert from XML to XSLX, use **Template** mode. In Table mode, the connector treats input XLSX files exactly like CSV files. The resulting output XML file has the following structure, where each row (record) in the original file becomes a child of a sheet wrapper element, which is a child of the root element **Items**. By default, the wrapper element is named after the worksheet. To control the name of the wrapper element, see [Root Element Name](#root-element-name). ```xml theme={null} ``` ### Template Mode In **Template** mode, the connector uses a template file to perform conversions. This template file matches the output file format: * When converting from XML to Excel, the template file is an Excel file. * When converting from Excel to XML, the template file is an XML file. These template files use ArcScript to dynamically populate the output files with data from the input files. For more information, see the [Templates](#templates) section. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Mode** | Specify which method (**Table** or **Template**) to use to translate the input file to the output format. See the [Overview](#overview) for more information. | | **Column headers** | (Table mode) Check this to have the translation use the values in the first row of the XLSX file as the element names of the child elements. If unchecked, the value elements are given generic names such as field\_0, field\_1, etc. | | **Template File** | (Template mode) The file that functions as the output template. Data is dynamically added to the file based on the scripting in the template. See the [Templates](#templates) section for details. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Sheet Names** | (Table mode) A comma-separated list of the workbook sheet names to output. By default, all sheets are output. | | **Start Row** | (Table mode) The first row in the sheet to use for the output table. | | **Start Column** | (Table mode) The first column in the sheet to use for the output table. | | **End Column** | (Table mode) The last column in the sheet to use for the output table. If not set, the column preceding the first empty cell on the first row is detected as the end of the sheet. | | **DateTime Format** | (Table mode) The format to use for DateTime values in the Excel document. | | **Number Format** | (Table mode) The format to use for numbers in the Excel document. Set this if you want all numbers in the document to be handled in the same way. | | **Root Element Name** | (Table mode) The XML element name to use for the wrapper element of each XML message output in Table mode. [Macros](#macros) are supported. The default, `%SheetName%`, names the element after the worksheet being processed. Use `%Sequence%` to include the processing sequence number of the sheet when processing multiple sheets. See [Root Element Name](#root-element-name) for details. | | **Propagate Merged Cell Values** | (Table mode) When reading Excel in table mode, check this to propagate the value of each merged cell to all rows in the merged area. This option provides more complete data but it requires an extra read of the Excel file, which can affect performance on large workbooks. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab *Settings related to the automatic processing of files by the connector.* #### Automation Settings | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Templates ### XML to Excel The following snippet from an XML file contains a list of elements under the `/Items/Orders` path. ```xml theme={null} PO0012345 Teddy 04/19/2023 23.98 Teddy Bear 14.99 Brown Truck 8.99 Red ``` To convert this XML file to XSLX, you must create an Excel template. The image below shows an example Excel template that uses *xmlDOMSearch* and *xpath* to loop through the elements of the XML file: ![Simple Template](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/simple_excel_template.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=707506e8770f8bccf2a0f13c61e5a992) Every Excel template must contain the following elements, which are shown in the example above: * Static column headers. In the example above, this is **Order Detail**. * Scripting in Excel notes. This scripting should use the [xmlDOMSearch](#xmldomsearch) operation to loop through the input XML at a specified XPath. In the example above, this is contained in the **arc:call** operation. * Scripting in Excel cells. This scripting should use the [xpath formatters](#xpath-formatters) to read values from the XML at a given xpath. This xpath is relative to the xpath specified in the *xmlDOMSearch* operation. The details of the *xmlDOMSearch* operation and *xpath* formatters in this example are described below. #### xmlDOMSearch The *xmlDOMSearch* requires two parameters: | Parameter | Description | | - | - | | **URI** | The resource path to the XML file to parse. The `[filepath]` attribute resolves to the input XML file to the connector, and the URI should almost always be set to this value. In the above example, the filepath is URL-encoded with the *urlencode* formatter to ensure that special characters in the filepath do not prevent the connector from reading the file: `[filepath \| urlencode]` | | **xpath** | The XML path to loop over in the document. The operation loops for each occurrence of the specified xpath. For example, with the xpath `/Items/Orders`, each *Orders* element that is a child of the root *Items* element causes a new set of output in the resulting Excel file. | #### xpath formatters The Excel notes with the *xmlDOMSearch* operation surround a block of cells that reference the input XML via the *xpath* formatter. The *xpath* formatter reads values from the input XML at the specified xpath. > [!NOTE] > > This *xpath* is relative to the path provided as a parameter to the *xmlDOMSearch* operation. In the above example, the first cell is populated with: `[xpath("OrderNo")]`. Since this xpath is relative, the cell is populated with the value from the following path in the input XML: `/Items/Orders/OrderNo`. If the *xmlDOMSearch* operation loops more than once—that is, if more than one instance of the operation's xpath parameter is found—then the block of cells between the Excel notes is repeated. The new cells are added vertically, so in the above example the second *OrderNo* cell would be directly below the first *Notes* cell. ### Excel to XML To convert from XSLX to an XML file, you must create a custom XML template using ArcScript commands. Since XML does not support the same scripting functionality as Excel files, the structure and formatting for converting from XSLX to XML is different than from XML to Excel. > [!NOTE] > > Due to the wide range of potential input and output structures, there is no universal solution for every conversion need. The example template shown below is just one of many potential solutions. You need to create a template based on your own needs. This example template uses a combination of standard XML syntax and Arc Script to read, process, and convert data from an input Excel file. The `arc:set` lines below specify the location of the file and the Excel version used to create the file. ```xml theme={null} ``` The sections below specify the conversion parameters and map the headers into standard XML formatting. The parameter `[_value | toalphanum | replace(' ','')]` indicates that the `arc:set` command should parse header names from the Excel file, convert them to alphanumeric format, eliminate blank spaces, and assign them as headers in the XML file. ```xml theme={null} ``` The sections below process the remaining data and build the data columns according to the specified structure. ```xml theme={null} <[_value]>[data.[_value]#[tmp.colIndex] | def] ``` ## Root Element Name In [Table mode](#table-mode), the connector outputs one XML message for each worksheet it processes. By default, the rows of each sheet are wrapped in an element named after the sheet. For a sheet named `Sheet1`, the output looks like this: ```xml theme={null} ``` > [!NOTE] > > If the sheet name changes from file to file, the wrapper element changes with it. Downstream [XML Map](./xml-map/xml-map) connectors that reference the sheet name then fail to match unless you use wildcards in the map. Use the **Root Element Name** setting to control the name of this wrapper element. The setting defaults to `%SheetName%`, which names the element after the worksheet being processed, as shown above. When a workbook contains multiple sheets, the connector outputs one message for each sheet. Use `%Sequence%` to include the processing sequence number of the sheet in the element name, so each message has a distinct wrapper element. For example, `Sheet_%Sequence%` produces `Sheet_1` for the first sheet processed, `Sheet_2` for the second, and so on. > [!NOTE] > > Element names must be valid XML names. After macros are evaluated, the connector silently replaces any invalid characters in the result with an underscore (`_`). For example, `Sales Data` becomes `Sales_Data`. ### Using a Static Name Enter a fixed value to give every output message the same wrapper element, regardless of the sheet name. For example, with a sheet named `Sheet1` and **Root Element Name** set to `Data`, the output is: ```xml theme={null} ``` This is useful when you only process the first sheet of a workbook, or when sheet names are not predictable. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Excel Operations In addition to the Operations provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except for two details: 1. They must be called through the `connector.rsc` endpoint. 2. They must include an auth token. For example, calling a connector operation using both of these rules might look something like this: ```xml theme={null} ``` Operations specific to the functionality of the Excel connector are listed below. ### excelClose Close an Excel connection. #### Optional Parameters * **handle**: The handle for the Excel file. #### Output Attributes * **success**: True if the connection is closed successfully. ### excelGet Queries the specified Excel worksheet. #### Required Parameters * **sheet**: The name of the Excel worksheet. #### Optional Parameters * **version**: The version of Excel you are using. The default is AUTO. You only need to choose another version if you are using a legacy Excel version. * **file**: The path to the Excel workbook. * **handle**: The handle for the Excel file. * **map:\***: This set of inputs contains a mapping of the attribute name and the name of the cell whose value is to be retrieved from the spreadsheet. For example, the attribute name map:MyValue which has a value of C1 pushes an attribute named MyValue with the value found in the cell at C1 in the sheet. You can specify a range of cell names to retrieve a range of cell values. #### Output Attributes * **\***: Depends on the content of the sheet and the query specified. If column headers are present they are used to name the output attributes. ### excelListSheets Lists the worksheets in a specified Excel workbook. #### Optional Parameters * **version**: The version of Excel you are using. The default is AUTO. You only need to choose another version if you are using a legacy Excel version. * **file**: The path to the Excel workbook. * **handle**: The handle for the Excel file. #### Output Attributes * **isHidden**: Returns true if the sheet is hidden from view in Excel. * **sheet**: The name of the Excel worksheet. Note that the name ends with \$. This is not required when specifying worksheet names to other operations. ### excelOpen Open an existing Excel workbook. #### Required Parameters * **file**: The path to the Excel workbook. #### Optional Parameters * **version**: The version of Excel you are using. The default is AUTO. You only need to choose another version if you are using a legacy Excel version. #### Output Attributes * **handle**: The handle which is used to execute other operations. # File Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/file The File connector pulls files from external directories into the flow and pushes files from the flow into external directories. The File connector can: * Pull files from external directories into the CData Arc flow * Push files from the Arc flow into external directories ## Key Capabilities * Local and network file system integration with UNC path support * Configurable authentication for protected network locations (.NET edition only) * Advanced filtering with glob patterns and recursive directory support * Temporary file handling with staging and rename operations for reliable transfers * Optional caching to ensure only new or updated files are downloaded ## Overview Each File connector is configured with a path on disk that determines what folder it reads files from or writes files to. This can be a local file path or a UNC path to access other file locations on the network. When a File connector receives a file from the configured path, it passes the received file off to the next connector in the Arc flow without modifying the file. In this way, File connectors can be used to poll an external folder location for files to pull into the Arc flow. When a file in the Arc flow is processed by a File connector, that file is written to the folder at the configured path. In this way, File connectors can be used to drop off processed files at a location external to the application. File connectors can be configured with username/password credentials to provide access to external filepaths that otherwise would not be accessible to the application. This allows Arc to pull files from protected folder locations without needing to grant the entire application access to the protected location. The path configured in the File connector can be dynamically populated using macros. For more information, see the [Macros](#macros) section. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Connector Settings *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Path** | The external filepath where the connector pulls files from or pushes files to. The path can include macros for dynamic evaluation, as described in [Macros](#macros). | #### Receive *Settings related to pulling files from the external filepath.* | Setting | Description | | - | - | | **File Mask** | A glob pattern for filtering files that should be pulled from the configured path. Only files matching the filemask are pulled. This setting can be combined with **Receive Filter** on the Advanced tab to specify multiple filters. | | **Delete files (after received)** | Whether to remove files from the external path after they are pulled into the Arc flow. When **Defer received-file finalization** is enabled, files are removed only after successful finalization. | | **Defer received-file finalization** | Keep received files pending until a downstream [Finalize](./finalize) connector completes them. Pending files are excluded from receiving again. **Receive Pending Timeout (minutes)** on the **Automation** tab releases them for another receive without deleting the source. This setting is disabled by default and does not affect sending files. For more information, see [Deferred Finalization](#deferred-finalization). | #### Authentication | Setting | Description | | - | - | | **Username** | The Windows user account that is used to access the files. If this is not specified, the connector uses the current user account. | | **Password** | The password for the specified user account. | | **Domain** | The domain for the Windows user account. This can be empty if the specified user is a local user account. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Max Receive Files** | The maximum number of files that are pulled in a single receive interval. If this is not a positive integer, no limit is applied. | | **Overwrite Option** | Specifies how the connector should handle the case when it attempts to write a file to the external path and that file already exists. The connector can *rename* the current file to a unique filename, *overwrite* the existing file, *append* the current file to the existing file, *skip* the operation without error, or *stop* the operation and throw an error. | | **Receiving Delay** | The amount of time (in seconds) that the connector waits before receiving files from the remote path. | | **Recurse** | Set this to true to download files in all subfolders of the target remote path. The directories are preserved for the received files. | | **Temp Send Path** | A directory to use as a staging folder before moving the file to its configured destination path. | | **Temp Send Prefix** | If specified, the connector uploads the file with the temporary prefix, then renames the file to its original filename after the operation is complete. | | **Temp Send Extension** | If specified, the connector uploads the file with the temporary extension, then renames the file to its original filename and extension after the operation is complete. | | **Minimum File Size** | The minimum file size to process, in bytes. If a file being uploaded is smaller than the specified size, it is finalized as **Skipped**. For files being downloaded, the connector adds a transaction log entry indicating that the file was skipped because of its size. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (e.g. \*.txt). Use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Separate multiple patterns by commas; later filters take priority except when an exact match is found. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | | **Receive** | A toggle that instructs the connector to automatically process files when they are ready and send them to the **Transactions** tab. | | **Interval** | The interval at which the connector processes all pending files and sends them to the **Transactions** tab. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service processes files at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before processing files. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | | **Receive Pending Timeout (minutes)** | The number of minutes a received file stays pending while it waits for a downstream [Finalize](./finalize) connector to complete it. When the timeout elapses, the message is marked with a Pending Timeout error, the source file is not deleted, and the file is released so that it can be received again. The default is 10 minutes. Set this to 0 to keep files pending indefinitely. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection File connectors must have the appropriate permissions to read/write from the configured path. Permissions issues are primarily a concern when the configured path is a UNC path to another server on the network, but can also arise when pushing or pulling files from a protected folder on the local disk. If the user running Arc does not have permission to access the path, set the **Username**, **Password**, and **Domain** fields in the [Authentication](#authentication) section of the **Settings** tab to a specific user with the appropriate permissions. ## Sending and Receiving Files ### Sending Files The File connector sends files from the **Transactions** tab to the external folder specified in the configured path. Files are automatically sent if **Send** is enabled on the [Automation](#automation-settings) tab. ### Receiving Files The File connector receives files from the external folder specified in the configured path and places them in the connector's **Transactions** tab. If the File connector is connected to another connector in the flow, the file does not remain in this folder and is instead passed along to the next connected connector. The connector automatically polls the external folder for files if **Receive** is enabled on the [Automation](#automation-settings) tab. The connector only pulls files that match the specified **File Mask**. If **Enable Timestamp Comparison** or **Enable File Size Comparison** is enabled, the connector caches file names pulled from the remote path and only receives files that are new or have been modified since they were last received. When multiple eligible files are available in a single receive operation, the connector processes them in ascending order by last-modified timestamp, so older files enter the flow first. Files with identical timestamps are processed in a consistent, predictable order. ### Deferred Finalization By default, the connector completes a file as soon as it receives it, and any configured cleanup, such as **Delete files (after received)**, happens immediately. If a later step in the flow fails, the source file is already gone. When **Defer received-file finalization** is enabled, the connector instead holds each received file in a pending state and waits for a downstream [Finalize](./finalize) connector to confirm that the flow succeeded. This is the recommended way to use **Delete files (after received)** when you need to keep the source file until processing is complete. To use deferred finalization: 1. Enable **Defer received-file finalization** on the File connector that receives the files. 2. Add a Finalize connector at the end of the success path of the flow. While a file is pending: * The connector does not receive the file again. * If the file reaches the Finalize connector, the connector completes the file and performs the configured cleanup. * If the file does not reach the Finalize connector before **Receive Pending Timeout (minutes)** elapses, the message is marked with a Pending Timeout error and the file is released so that it can be received again. The source file is not deleted. The connector records a **Send Failure: Pending Timeout** error notification. * If the source file changes after it was received, the connector does not delete it, even when the file is finalized. Deferred finalization applies only to receiving files. It does not change how the connector sends files. ### Subfolder Headers If **Recurse Subdirectories** is set to **True**, when receiving a file: * The message for the received file contains a Subfolder header in its metadata. * This header contains the subfolder, relative to the configured path, that the file was received from. * This subfolder header is supported by other connectors that support a Subfolder header in Send operations. When sending files, if a Subfolder header is present on the message that is sent to the configured path in the File connector: * The file is placed in the subfolder (relative to the path setting in the File connector) that is specified in the subfolder header. * The subfolder is created if permissions are available. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Flat File Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/flat-file The Flat File connector converts flat file formats into XML and generates flat files from XML, supporting position-delimited and character-delimited formats. Flat File connectors convert flat file formats into XML, and vice versa. ## Key Capabilities * Bidirectional flat file/XML parsing for position-delimited and character-delimited formats * Support for multiple line types with control field identification * Automatic hierarchy detection and XML nesting for master-detail relationships * Comprehensive field mapping with custom data type handling ## Overview Each Flat File connector is configured with a specific flat file format to convert to XML or to generate from XML. The Flat File connector has two primary modes: * Position Delimited, where the connector is configured with an arbitrary set of field names, indices (such as positions), and lengths indicating where data appears in each line of the flat file. * Character Delimited, where the connector is configured with the character that separates field values in the flat file. See [Defining the Flat File Format](#defining-the-flat-file-format) for details. > [!NOTE] > > You might also need a numeric control field. Follow the instructions in [Non-Standard XML Control Fields](#non-standard-xml-control-fields) to set this up. The Flat File connector supports defining multiple types of lines in the flat file. For example, a flat file might have a *header* line representing Purchase Order data, and multiple *item* lines representing the Line Items in the order. The key to defining multiple line types is to specify the **Control Field**; this value is what determines the type of a specific line in the flat file data (for example, a header line might have a control field value of HEAD while an item line has a control field value of ITEM). See [Multiple Line Types](#multiple-line-types) for details on how to configure multiple line types. Once the flat file format is configured, the connector converts files that match this format into XML. The resulting XML structure is explained in [XML Format](#xml-format). The Flat File connector can also translate XML that fits this structure into a flat file according to the defined flat file format. Some flat files have implied hierarchical relationships between different line types. For information on preserving these hierarchical relationships when converting the flat file to XML, see [Multi-Line Hierarchy](#multi-line-hierarchy). ## Connector Configuration ### File Details Tab | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **File Type** | **Position Delimited**—the fields in the flat file appear at a specific position in each line. **Character Delimited**—the fields in the flat file are separated by a specific character defined by the **Delimiter** field. | | **Delimiter** | If **File Type** is **Character Delimited**, specify the character that separates individual fields in the flat file. | #### Control Field: Position Delimited *Settings related to the Control Field when the connector is set to Position Delimited.* | Setting | Description | | - | - | | **Multi-line Mode** | Whether the flat file format includes multiple line types. See [Multiple Line Types](#multiple-line-types) for more information. | | **Start Index** | If **Multi-line Mode** is enabled, this value is the index in the line where the **Control Field** begins. For example, if the first field in a line is the control field, then the Start Index should be 0. | | **Control Field Length** | (Optional) If **Multi-line Mode** is enabled, this value defines the number of characters to read from the **Start Index** to get the value of the **Control Field**. | | **Column Headers Present** | If **Multi-line Mode** is disabled, this determines whether the first line of a flat file should be interpreted as column headers. Enabling this setting causes the connector to generate a header row when converting from XML to flat file. | #### Control Field: Character Delimited *Settings related to the Control Field when the connector is set to Character Delimited.* | Setting | Description | | - | - | | **Multi-line Mode** | Whether the flat file format includes multiple types of lines. See [Multiple Line Types](#multiple-line-types) for more information. | | **Field Index** | If **Multi-line Mode** is enabled, this is the index of the **Control Field** in the line (indices start at 0). For example, if the second field of a line is the control field, then the Field Index should be 1. | | **Generate Field/Line Type Names** | If **Multi-line Mode** is enabled, this setting provides the option to *not* specify the names and indexes of the fields and line types in the flat file. When this setting is enabled, the connector auto-generates XML element names for fields and line types that are not explicitly defined in the [Line Types](#line-types) section below. | | **Column Headers Present** | If **Multi-line Mode** is disabled, this determines whether the first line of a flat file should be interpreted as column headers. Enabling this setting causes the connector to generate a header row when converting from XML to flat file. | | **Generate Field Names** | If **Multi-line Mode** is disabled, this determines whether the connector generates generic field names. Disable this setting if you want to manually specify the field names in the **Line Types** section. | #### Line Types This section lets you define the field names and positions in the flat file format. See [Defining the Flat File Format](#defining-the-flat-file-format) for details on defining the flat file format. If **File Type** is **Position Delimited** and **Multi-line Mode** is enabled, click the **Add Line Type** button to define multiple line types. Each line type has a **Control Field Value**, which is used to identify the line type. For example, a header line might have the control field value of HEAD, while an item line has the control field value of ITEM. If **File Type** is set to **Character Delimited**, **Multi-line Mode** is enabled, and **Generate Field/Line Type Names** is enabled, you do not need to provide names and indexes for all of the fields or line types present in the flat file. In this case, the connector auto-generates XML element names for any unspecified fields or line types. Uncheck **Generate Field/Line Type Names** to manually specify the field names. ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Padding Character** | When creating a flat file and the field value does not fill the entire field length, this character is used to fill the rest of the field. | | **Invalid XML Name Prefix** | Some field names are not valid names for XML elements (for example, fields that begin with a number like '123ABC'), so you must apply a prefix to generate XML from the flat file. When translating from XML to flat file, the connector looks for this prefix and removes it. | | **Line Separator** | Specifies the control character that the connector uses for line endings when writing files. Options are **LF** (default) and **CRLF**. | | **Nest Line Types** | This setting is only relevant when converting flat files to XML, and when there are multiple line types in the flat file. If **Multi-line Mode** is enabled, enabling this setting instructs the connector to add hierarchy to the resulting XML based on the control fields of the flat file lines. See [Multi-Line Hierarchy](#multi-line-hierarchy) for details on this automatic hierarchy. | | **Pad on EOL** | By default, the connector throws an error if it encounters an unexpected end-of-line. When enabled, this setting tells the connector to pad out the remainder of the line instead of throwing an error. | | **Force Cell Delimiter** | If **File Type** is set to **Character Delimited**, enabling this setting instructs the connector to wrap each cell value in cell delimiters (`"`). Disabling this setting instructs the connector to only wrap the cell value in cell delimiters if the specified character delimiter is present in the cell value. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Defining the Flat File Format The first step in configuring a Flat File connector is defining the format of the flat file. This section describes formats that have a single line type; in other words, each line in the flat file has the same set of fields. For flat files with multiple line types, see [Multiple Line Types](#multiple-line-types). ### Character Delimited Single-line Format For Character Delimited flat files, defining the format is simple: use the **Delimiter** property to specify the character that separates different fields in the flat file. The **Column headers present** field indicates whether the first line of the flat file is a header line; in other words, it contains the names of fields rather than actual data. If these column headers are present, the connector uses the header names as the names of XML elements in the resulting translated XML. Similarly, the connector uses the XML element names to generate a header row when translating from XML to flat file. If column headers are not present, the connector supports manually specifying the names of each field by adding the fields in the [Line Types](#line-types) portion of the **Settings** tab. These field names are applied in index order, meaning that the first entry in **Line Types** is the name of the first field in the flat file line, and so on. You can also auto-generate generic field names by enabling **Use auto generated field names**. ### Position Delimited Single-line Format For Position Delimited flat files, defining the format requires specifying the position of each field in the format. The [Line Types](#line-types) portion of the **Settings** tab provides an interface for adding an arbitrary number of fields that are present in each line of the flat file. Each field must be identified by a name and the position in the file it appears. The **Column headers present** field indicates whether the first line of the flat file is a header line; in other words, it contains the names of fields rather than actual data. The field names must still be configured in the **Line Types** section; this setting simply helps ensure that a header row is not interpreted as real data. ### Non-standard XML Control Fields Flat file data can include lines that contain a control field that is fully numeric or begins with a number. While this is normal in flat file data, it poses a challenge when converting that flat file data into XML, because XML elements cannot start with a number. They must begin with a letter or an underscore. To accommodate this in the Flat File Connector, if a control field starts with a number, Arc automatically prepends two underscores (\_\_) to that value in the output XML. In addition, a `lineType` attribute is added to that element and its value corresponds to the original control field value. For example, the following flat file data includes two line types with control fields of `000` and `111`: ``` 000 12345 01222025 NC Yes 000 02468 01242025 MD Yes 111 82390 01202025 FL No 111 67524 01132025 CO No ``` The flat file configuration for this data would look something like this: ![](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/flat_file_control_fields.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=6ec73843f736e27afce5ba17dbd6de6e) When this flat file data is run through the connector using this configuration, the output looks like the following: ```xml theme={null} <__000 lineType="000"> 12345 01222025 NC Yes <__000 lineType="000"> 02468 01242025 MD Yes <__111 lineType="111"> 82390 01202025 FL No <__111 lineType="111"> 67524 01132025 CO No ``` Notice that each line type has been prefixed with two underscores, and there is a `lineType` attribute for the XML elements that correspond to the original control field value. This allows the flat file data and XML to be fully bi-directional, which means that when output XML is passed as input to the flat file connector, the output is flat file data containing the correct line type definitions without any underscores. It also means that if you are going from XML to flat file and you need a control field for a line type beginning with a number (or is all numbers), you need to implement the `lineType` XML attribute on the appropriate XML elements in your document mapping, prior the document arrives at the Flat File connector. ## Multiple Line Types If your flat file format contains multiple types of lines, enable the **Multi-line Mode** property in the **Control Field** portion of the **Settings** tab. The field in the flat file that identifies the line type is known as the control field. ### Character Delimited Multi-line Format When the **File Type** is Character Delimited, the **Field Index** setting determines where the control field appears in each line of the flat file. This index starts at 0, meaning that if the control field is the 5th value in the line, the **Field Index** should be 4. For each possible value that might appear in the control field define a new line type by clicking the **Add Line Type** button in the [Line Types](#line-types) portion of the **Settings** tab. Set the value that identifies the line type in the **Control Field Value** for that line. Once each possible line type has been added and identified using a specific **Control Field Value**, the fields that appear in each line type should be specified in index order. If **Generate Field/Line Type Names** is enabled, you do not need to provide names and indexes for all of the fields or line types present in the flat file (only the **Control Field** is required). In this situation, the connector auto-generates XML element names for any unspecified fields or line types. ### Position Delimited Multi-line Format When the **File Type** is Position Delimited, the **Start Index** setting determines the position where the control field begins in each line of the flat file. This index starts at 0, meaning that if the control field begins at the 15th character in the line, the **Field Index** should be 14. For each possible value that might appear in the control field, define a new line type by clicking the **Add Line Type** button. Set the value that identifies the line type in the **Control Field Value** for that line. Once each possible line type has been added and identified using a specific **Control Field Value**, the fields that appear in each line type should be specified along with the position in the line where they begin. ### Multi-line Example For example, say that a flat file contains two types of lines, a shipment line type and a package line type. The shipment line contains information on the date, time, and address of shipment, and the package line type contains information on the items being shipped. The first field in each line is SHIP or PCKG to indicate what type of line it is. To handle this case, enable **Multi-line** mode, and set the **Field Index** or **Start Index** to 0 to indicate that the control field is the first field in the line. Then, configure two line types in the **Line Types** section: one with a **Control Field Value** of SHIP, which contains each field in a shipment line (for example, ShipDate, DeliveryDate, and ShipToAddress), and one with a **Control Field Value** of PCGK, which contains each field in a package line (such as ItemName and ItemWeight). ## XML Format Once a flat file is converted to XML, the result has the following XML structure: * One `Items_` element at the root of the document * Each line in the flat file has an element named the same as the **Control Field Value** for that line (or row if no **Control Field Value** is defined) * Each field in the row is a child element of the **Control Field Value** element For example, if a flat file has SHIP and PCKG lines, then the XML output would look similar to this: ```xml theme={null} 12B992 20230228 14 Wallaby Way 12B992 Goggles 3.98 12B992 Fins 1.07 ``` To convert an XML file into a flat file, the XML input must match the structure above (this includes the restriction that the field names must match the defined fields in the connector configuration). When converting XML into a flat file, the connector creates a new line in the resulting flat file for each row element in the input XML. For each child of the row element, the connector matches the child to a field name in the connector configuration, and places the value for that element at the appropriate field index. ## Multi-Line Hierarchy Flat files with multiple line types often have an implied hierarchical relationship between the line types. For example, one line might contain data for a Purchase Order (such as customer name and order date), while the following lines represent each line item in that order (for example item name and item quantity). The line items *belong* to the order, creating a hierarchical relationship. These relationships are sometimes referred to as master-detail relationships, which in our example means that the first line type (Purchase Order) is the master, and the following lines (line items) are details about that master line. In XML, this is often represented as a parent-child relationship. The Flat File connector can preserve these hierarchical relationships when translating flat files to XML. If you enable the **Nest Line Types** setting on the [Advanced](#advanced-tab) tab, the connector automatically indents the detail line types so that they are children of the master line types. The following is a simple example of translating a hierarchical flat file with **Nest Line Types** enabled. The section after the example explains the logic the connector uses to determine the hierarchical relationships. ### Multi-Line Hierarchy Example Take as example input the following flat file: ``` M,CustomerA,05022020,true D,itemA,4,2.09 D,itemB,9,15.23 M,CustomerB,05032020,false D,itemC,1,5.99 D,itemE,1,3.99 ``` There are two line types in this flat file, `M` and `D`: * `M`, or master, contains information for an order: customer name, order date, and a high-value customer flag. * `D`, or detail, contains line item information for the preceding order: item name, item quantity, and item price. Since the `D` line type *belongs* to the `M` line type in a hierarchical relationship, you might want to preserve this hierarchy when translating the flat file to XML. To see what this looks like, take the following example output when the **Nest Line Types** setting is enabled and the flat file above is translated to XML: ```xml theme={null} CustomerA 05022020 true itemA 4 2.09 itemB 9 15.23 CustomerB 05032020 false itemC 1 5.99 itemE 1 3.99 ``` Notice that the `D` records are children of the `M` records, which accurately reflects the hierarchy present in the source data. ### Multi-Line Hierarchy Logic The Flat File connector uses the order of lines in the source flat file to determine the hierarchy between line types. When **Nest Line Types** is enabled, the connector uses the following logic: * The first line type the connector encounters is always treated as the top level of hierarchy (it is not indented in the resulting XML) * After the first line type, when the connector encounters a new line type it assumes this line type is one level of hierarchy below the previous line type (it *belongs* to the previous line and is therefore indented in the resulting XML) * When the connector encounters a line type it has encountered before, it returns to that line type's level of hierarchy, closing all XML elements at hierarchy levels equal-to or below this level. This means that the source flat file must have lines in order according to the desired hierarchy. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Form Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/form Form connectors expose a public web endpoint where clients fill out a form and submit data into a flow, with a drag-and-drop designer and user authentication support. Form connectors expose a public web endpoint where clients fill out a form and submit that data into a CData Arc flow. ## Key Capabilities * Web-based form creation with drag-and-drop designer * Support for sections, detail tables, and multiple field types * User authentication and access control for form submissions * Dynamic form generation with customizable branding and success messages ## Overview Forms are created on the connector **Form Designer** tab. A form is divided into sections and detail tables, each of which can have a variable number of fields. For details, see [Creating Forms](#creating-forms). When you use a Form connector to submit new data, you should place the connector at the start of a connected flow. Clients can connect to the web endpoint, view and fill out the form, and submit this data into the configured Arc flow. For more information, see [Submitting Form Data](#submitting-form-data). The following sections explain how to configure the connector, create forms, and submit new form data. ## Connector Configuration This section contains all of the configurable connector properties. ### Form Designer Tab *Settings related to building a set of form fields that are organized into sections and detail tables.* ![](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/Form_connector_form_designer.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=955e4c4c928c44d8b99d3654dccb8584) When clients connect to the configured endpoint and submit a form, they see fields that are laid out exactly as they are configured in the panel above. The designer includes the following settings (see [Creating Forms](#creating-forms) for more details). | Setting | Description | | - | - | | **Form URL** | A link to the published form. | | **Form** | Title of the form (for example, your company name). | | **Add** | Dropdown to add a section or a detail table. **Add Section** lets you add specific sections to a form, and **Add Detail** lets you add a detail table to the form. Detail tables act like a pre-configured table where the form user can add as many records as they need for the dataset. | | **Form Fields** | Drag and drop fields into detail table and section areas of the form. Repeat this as many times as necessary. The field types are: **Text Field** — An input text box with a name and description. **Dropdown** — A dropdown list with name and dropdown options. **Radio Buttons** — A set of radio buttons with name and radio button options. **Checkboxes** — A set of checkboxes with name and checkbox options. **File Upload** — Allows form users to upload specific file types directly to the form. | | **Logo** | Upload your company logo, which appears at the top of the form. By default it uses the CData Arc logo. You can also change the color of the banner. | | **Success Message** | The confirmation message users see after they submit form data successfully. You can change the default message or add additional text. | ### Settings Tab #### Connector Settings *Settings related to the core configuration of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Form Users Tab The **Form Users** tab lets you create user credentials for clients so that they can access the public endpoint. If a client is not logged into the application administration console already, that client is prompted to enter the **User** and **Password** combination when they browse to the endpoint. You can add, edit, and delete credentials on this tab. #### Adding Users To add user credentials: 1. Click **Add User** to open the **Add Form User** dialog box. 2. Enter a username and password. (Click the eye icon to see the contents of the password field.) 3. Click **Save Changes** and the new user is added to the list of form users. #### Editing Users To edit users: 1. Click the ellipses next to the user that you want to edit, then click **Edit**. 2. Edit the credentials and click **Save**. If you decide not to make changes, click **Cancel**. #### Deleting Individual Users To delete individual users: 1. Click the ellipses next to the user that you want to edit, then click **Remove**. 2. Click **Remove** to delete the user or **Cancel** to cancel the deletion request. #### Deleting Multiple Users To remove multiple users, select the checkboxes for the users you want to delete, then click **Remove**. Confirm the deletion, or cancel the request. ### Server Tab #### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature:
  • `localhost` cannot be modified or removed from the list.
  • Any IP addresses outside of the defined ranges are rejected.
  • Ranges are supported. For example, the entry `100.10.100.1-15` indicates that IP addresses between `100.10.100.1` and `100.10.100.15` are allowed.
  • Classless inter-domain routing (CIDR) notation is supported. For example, the entry `100.10.100.0/24` indicates that IP addresses between `100.10.100.0` and `100.10.100.255` are allowed.
  • Wildcard patterns are supported. For example, the entry `100.10.100.*` indicates that IP addresses beginning with `100.10.100` are allowed.
> [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. ### Advanced Tab #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Incoming form responses appear on this tab. ## Configuring the IIS Server If your Arc instance is hosted on an IIS server and you use Microsoft Windows authentication to login to the application, you must disable Windows authentication for your users. Otherwise, the Form connector cannot properly authenticate users, and the connector will not work properly. To disable Windows authentication: 1. In your IIS manager, create a new folder named `connector` in the base application path. 2. Select **Authentication** under **IIS** (in the right pane) for the `connector` folder. ![](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/form_iis_1.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=755424febc90c78895ef6e58bd672e05) 3. Set **Windows Authentication** to **Disabled** for this folder. ![](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/form_iis_2.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=4e18e1e61bead62351853214ad09b0dd) ## Creating Forms Forms have fields, sections, and details: * **Section**: A heading that groups fields together * **Detail**: An object that acts like a pre-configured table where the form user can add as many records as they need for the dataset * **Field**: An input box with a label where users enter form data To get started, click the **Add** button next to the form title, and choose whether to **Add Section** or **Add Detail**. ### Adding Sections *Sections* divide fields in the form by categorizing them under a heading. For example, the **Ship To** section might contain fields like **Address**, **City**, and **Zip**. Sections are not required in order to add fields to the form. However, sections provide logical groupings of items that help users quickly understand the form. To add a section: 1. Choose **Add Section** from the dropdown. Enter a section name. 2. Now you can start [Adding Fields](#adding-fields) to the section. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/Form_connector_form_designer_add_section_results.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=ea94ffb288c3fa4bca91275cea39ea4d) ### Adding Detail Tables *Detail* tables allow form creators to embed a table in the web form to enable structured, row-based data entry. They include a configurable number of columns which you define during the form creation. They are ideal for capturing repeatable datasets in a single form submission. End users can then dynamically add as many rows as needed and enter values for each column in those rows. See [Accessing Your Form](#accessing-your-form) for an example. To add a detail table to your form: 1. Click **Add Detail** from the dropdown. Enter a name. 2. Now you are ready to add columns to the table. Select any of the [form field types](#form-field-types). ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/Form_connector_form_designer_add_detail_result.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=26f04e1dcdcafe49f6ee1f551b243b3c) ### Adding Fields *Fields* are the basic unit of input for the form. To add a field: 1. Drag an object from the **Form Fields** section into your section or detail table. 2. Enter a **Display Name** (for example, Line Item Id). By default, **XML Name** is populated, but you can use the **Override** checkbox to supply a different name. 3. Use the **Is required** checkbox to specify whether the field is mandatory. 4. The other options vary based on the [form field type](#form-field-types). The following example shows a text field being defined. When you are finished defining the field, click **Add**. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/Form_connector_form_designer_form_field.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=09558d235c1ad8a2a18fbc6f65fcacec) > [!TIP] > > To edit a field, click the link to its name, or the ellipses to the right of the name.To remove a field, click the ellipses and choose **Remove Instance**. Drag and drop fields in a section or detail table to reorder their position in the form. ### Form Field Types #### Text Field Options * Display Name: Name of the field in the form. * XML Name: Sanitized version of the Display Name. It must be a valid XML element name. * Override: Checkbox which allows you to change the default XML Name. * Is required: Checkbox indicating whether the field is mandatory. * Default Value: Populates the form with the text supplied here. * Description: Helper text describing what the field is for. #### Dropdown Field Options * Display Name: Name of the field in the form. * XML Name: Sanitized version of the Display Name. It must be a valid XML element name. * Override: Checkbox which allows you to change the default XML Name. * Dropdown Options: The list of options that appear in the form dropdown field. Click **Add Option** to add the options you need. * Is required: Checkbox indicating whether the field is mandatory. * Default Value: Populates the form with the text supplied here. * Description: Helper text describing what the field is for. #### Radio Button Field Options * Display Name: Name of the field in the form. * XML Name: Sanitized version of the Display Name. It must be a valid XML element name. * Override: Checkbox which allows you to change the default XML Name. * Radio Button Options: The list of options that appear in the form radio button field. Click **Add Option** to add the options you need. * Is required: Checkbox indicating whether the field is mandatory. * Default Value: Populates the form with the text supplied here. * Description: Helper text describing what the field is for. #### Checkbox Field Options * Display Name: Name of the field in the form. * XML Name: Sanitized version of the Display Name. It must be a valid XML element name. * Override: Checkbox which allows you to change the default XML Name. * Checkbox Options: The list of options that appear in the form checkbox field. Click **Add Option** to add the options you need. * Is required: Checkbox indicating whether the field is mandatory. * Default Value: Populates the form with the text supplied here. * Description: Helper text describing what the field is for. #### File Upload Field Options * Display Name: Name of the field in the form. * XML Name: Sanitized version of the Display Name. It must be a valid XML element name. * Override: Checkbox which allows you to change the default XML Name. * Accepted File Type Extensions: The allowed list of upload file types. For example, `.xml` only accepts XML files. To allow all file types, use `*`. Click **Add Option** to add the options you need. * Is required: Checkbox indicating whether the field is mandatory. * Default Value: Populates the form with the text supplied here. * Description: Helper text describing what the field is for. ### Accessing Your Form When you finish designing your form, access it by clicking the **Form URL** link at the top of the designer. The following form is a simple example: ![](https://mintcdn.com/cdata-arc/gRZIlTTCd5leT0Io/26.4/images/Form_connector_final_form.png?fit=max\&auto=format\&n=gRZIlTTCd5leT0Io\&q=85\&s=549edfe220d293fee8dce4e537db4e0a) ## Submitting Form Data After you create a form, users access the Form connector's web endpoint and submit data using the form. For fields in the main form, users simply enter the values that correspond to the labels on the form (which are taken from the name of the fields that you created). Users can add rows to the detail table by clicking **Add Row**. Once a user enters all their data, they click **Submit** to submit their data. If the submission is successful, a message similar to the following appears. ![](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/Form_connector_form_designer_success_message.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=2e27cc8b5c15eedfe3d49d3c68e62e0e) When a user submits a form, the form data is pushed to the **Transactions** tab. The output is an XML file that contains the following structure: ```xml theme={null} data more data even more data 123 456 135 246 ``` This output is well suited to being processed by an [XML Map Connector](./xml-map/xml-map) so that it can be mapped into an XML format that is usable by another connector. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# FTP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ftp Configuration and usage guide for the CData Arc FTP connector, which transfers files to and from a remote FTP server. The FTP Connector transfers files to and from a remote FTP server. ## Key Capabilities * File transfer protocol support with FTPS (FTP over TLS/SSL) and traditional FTP * Active and passive mode support with comprehensive firewall compatibility * Resume capability for interrupted uploads and downloads * Advanced authentication including certificate-based client authentication * Connection pooling and temporary file handling for reliable transfers * Optional caching to ensure only new or updated files are downloaded ## Overview Each FTP Connector connects uploads to a specified folder and downloads from a set of folders on the remote FTP server. Files that reach the FTP Connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder(s) are downloaded and entered into the Arc flow. The connector supports upload and download filters to only send and receive specified file names and/or file extensions. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to establishing the connection to the remote FTP host.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Remote Host** | The hostname or IP address of the FTP server. | | **TLS Type** | How to negotiate TLS/SSL when connecting to the server. Choose **Explicit** to establish a plain text connection where TLS/SSL is then started with an explicit command. Choose **Implicit** to immediately negotiate TLS/SSL without establishing a plain text connection. | | **Remote Port** | The port on which to connect to the FTP server. | #### Client Authentication *Settings related to authenticating to the FTP server.* | Setting | Description | | - | - | | **User Name** | The username for logging in to the FTP server. | | **Password** | The password for logging in to the FTP server. | #### Server Authentication *Setting related to verifying the FTP server's identity.* | Setting | Description | | - | - | | **Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. | #### Upload *Setting related to uploading files to the FTP server.* | Setting | Description | | - | - | | **Remote Path** | Files processed by the connector are uploaded to the specified remote folder. You can use macros to dynamically create dated folders (such as year or month folders) in your path. For more information, see [Macros](#macros). | #### Download *Settings related to downloading from the FTP server.* | Setting | Description | | - | - | | **Remote Path** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. You can use macros to dynamically create dated folders (such as year or month folders) in your path. For more information, see [Macros](#macros). | | **File Mask** | A glob pattern that determines which files within the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete files (after received)** | Check this to delete successfully downloaded files from the FTP server. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Client Certificate** | The private certificate presented during TLS client authentication. | | **Client Certificate Password** | The password required to access the TLS client certificate. | #### Server Commands These options allow you to execute custom commands on the target server at various times during connector execution. > [!NOTE] > > While the commands described below are commonly accepted, be sure to check with the target server administrator to ensure that the server can accept a command before trying to use it.
SettingDescription
**After Connect**Execute this command directly after connecting.
**After Get**Execute this command after downloading each file.
**After Put**Execute this command after uploading each file.
**Before Get**Execute this command before downloading each file.
**Before Put**Execute this command before uploading each file.
Following are the commands you can submit to the target server when you include them in the fields described above: • `onerror`: Pass the `fail` argument to instruct the connector to treat errors that come from the custom scripts as errors for the overall transaction. Without `fail` the connector logs any exceptions and continues processing. • `cd`: Change directory. • `mkdir`: Create directory. • `rn` or `move`: Rename or move file. • `del` or `rm`: Delete or remove file. • `rmdir`: Remove directory. CData Arc supports the following macros with these commands. They all use the following syntax: `%Macro%`. Some macros, such as `%Date%` and `%SourceFilename%`, do not require an argument, but others do. The macros that take an argument use the following syntax: `%Macro:argument%`.
MacroDescription
MsgHeader:*headername*Evaluates to the header value of the message (for example: %MsgHeader:headername%).
SourceFilenameEvaluates to the filename (extension included) of the file currently being processed.
SourceFilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed.
DateEvaluates to the date the file was last modified in a yyyy-MM-dd format (for example, 2024-02-13).
DateTimeEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss format (for example, 2024-02-13T08:14:22).
TimeEvaluates to the time the file was last modified in a HH:mm:ss format (for example, 08:14:22).
DateTime.tzEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss zzz format (for example, 2024-02-13T08:14:22 PST).
Time.tzEvaluates to the time the file was last modified in a HH:mm:ss zzz format (for example, 08:14:22 PST).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
**Example**: After downloading, you want to move each file from the download folder to a History subfolder. Use the **After Get** field to supply the `move` command with the `%SourceFilename%` macro. ![Server command example](/26.4/images/server_command_example.png) #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Account** | The Windows user account whose credentials should be presented to the FTP server. | | **Active Mode** | Check this to enable Active mode on the remote FTP server. Keep in mind that Passive mode is less likely to result in firewall interference with the connection. | | **Clear Command Channel** | Check this to send FTP commands using a cleartext channel instead of an encrypted channel. | | **Clear Data Channel** | Check this to have the application use a clear data channel when communicating with the FTP server. | | **Use Connection Pooling** | Check this to have the connector utilize a pool to access and store connections. This is useful in scenarios where the server places restrictions on the frequency of logon attempts. | | **Download Excluded Extensions** | A comma-delimited list of file extensions that should not be downloaded by the connector. | | **Download Excluded Prefixes** | A comma-delimited list of prefixes that should not be downloaded. The connector skips filenames that begin with these prefixes. | | **EPSV** | Check this to use extended passive mode. This might be necessary if you need to use a protocol other than IPv4. | | **FSwitch** | Used in conjunction with **Simple Dir List**. Check this to restrict the directory listing to files only. | | **Max Download Files** | The maximum number of files to be downloaded from the SFTP server in one polling interval. Set to **-1** to allow all files to be downloaded. | | **Minimum File Size** | The minimum file size to process, in bytes. If a file being uploaded is smaller than the specified size, it is finalized as **Skipped**. For files being downloaded, the connector adds a transaction log entry indicating that the file was skipped because of its size. | | **Maximum File Size** | The maximum file size to process, in bytes. If a file being uploaded is larger than the specified size, it is finalized as **Skipped**. For files being downloaded, the connector adds a transaction log entry indicating that the file was skipped because of its size. | | **Monitor File Growth** | Check this to have the application store the file sizes of the remote files between polling intervals, and only download files that do not change in size. This prevents the connector from downloading files that are still in the process of being written on the server. | | **Passive Connection Address** | Only used when **Active Mode** is not checked. If unchecked, the application parses the remote host to send replies from the previous server response. If you check this, the application uses the value of the **Remote Host** setting. | | **Prot for Implicit TLS** | Tells the server that the data channel is protected for implicit TLS connections. Select the TLS type on the **Settings** tab. | | **Recurse** | Check this to download files in subfolders of the target remote path. The subdirectories are preserved for the received files. | | **Resume Downloads** | If enabled, when an error occurs during a file download, the connector stores the portion of the file that was downloaded and attempts to download the remainder of the file on the next download attempt. | | **Resume Uploads** | If enabled, when an error occurs during a file upload, the connector notes the number of bytes that were transferred and attempts to upload the remainder of the file on the next upload attempt. | | **Simple Dir List** | Check this when you need a simple directory listing command for servers that send back long responses that cannot be parsed. After you set this, if the connector is still unable to parse the directory listing, add the **FSwitch** setting to restrict the directory listing to files only. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Keep in mind that SSLv2 and SSLv3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support them. | | **Temp Download Extension** | A temporary file extension to append to file downloads while the transfer is in progress. | | **Temp Upload Extension** | A temporary file extension to append to file uploads while the transfer is in progress. | | **Temp Upload Path** | A temporary path on the remote FTP server where files are uploaded. The file is moved into the permanent upload path after the transfer is complete. | | **Temp Upload Prefix** | A temporary file prefix to prepend to file uploads while the transfer is in progress. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Transfer Mode** | Whether files should be transferred as ASCII or binary. If set to **Default**, the server chooses the transfer mode. | | **Upload Single Transaction** | Check this to upload all files in a single transaction. By default (unchecked), the application logs off from the server after each upload. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Use MLSD for directory listings** | Check this to use the MLSD command to return detailed directory listings. This might not be supported on legacy FTP servers, which might return an error such as *Command not understood*. Uncheck this to address that error on legacy servers. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Upload** | Whether files arriving at the connector should automatically be uploaded. | | **Retry Interval** | The number of minutes before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connect retries the file indefinitely. | | **Download** | Whether the connector should automatically poll the remote download path(s) for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Connecting to the remote FTP server requires the following settings: * Remote Host * TLS Type * Remote Port * User Name * Password FTP servers might also require **Server Public Certificate** (on the **Settings** tab) and/or [TLS Client Authentication](#tls-client-authentication) (on the **Advanced** tab). ## Uploading Files The FTP Connector uploads files from the **Transactions** tab. Files are automatically uploaded to the FTP server if **Upload Automation** is enabled in the connector. **Remote Path** should be set to the folder on the remote FTP server where files should be uploaded. You can use [macros](#macros) to dynamically create dated folders (such as year or month folders). The application changes directories to the specified folder after logging in to the server. You can set a temporary upload extension, prefix, or path on the **Advanced** tab to ensure that files have been fully transferred before they end up in the **Remote Path** folder. If any files could not be uploaded, the connector attempts to send them again after the **Retry Interval** minutes have elapsed. This process continues until the **Max Attempts** setting is reached, after which the connector raises an error and the file is no longer sent automatically. ## Downloading Files Downloaded files are displayed in the **Transactions** tab of the connector, or passed along to the next connector in the flow. Files are automatically downloaded from the FTP server if **Download Automation** is enabled in the connector, and can be manually downloaded by clicking **Receive Files** button on the **Transactions** tab. **Remote Path** should be set to the folder on the remote FTP server from which files should be downloaded. You can use [macros](#macros) to dynamically create dated folders (such as year or month folders). The application changes directories to the specified folder after logging in to the server. The **File Mask** property filters files so that only filenames matching the glob pattern are downloaded from the server. You can set a temporary download extension or prefix on the **Advanced** tab to ensure that files have been fully transferred before they end up in the **Transactions** tab. You can use the **Delete files (after download)** setting to delete the files on the remote server after successfully downloading them. After configuring, files are downloaded according to the **Receive Automation** settings, or by manually clicking **Receive Files** on the **Transactions** tab. ## Subfolder Headers If **Recurse** is checked on the **Advanced** tab, when receiving a file from the **Remote Path** in the FTP connector: * The message for the file that is received contains a Subfolder header in its metadata. * This header contains the subfolder, relative to the **Remote Path** in the FTP connector, that the file was received from. * This subfolder header is supported by other connectors that support a subfolder header in Send operations. When sending files, if a Subfolder header is present on the message that is sent to the **Upload Path** in the FTP connector: * The file is placed in the subfolder, relative to the **Upload Path** in the FTP connector, that is specified in the subfolder header. * The subfolder is created if permissions are available. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# FTP Server Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ftp-server Configuration and usage guide for the CData Arc FTP Server connector, which provides connection credentials to access the built-in FTP server. FTP Server connectors provide connection credentials to access the full-featured Arc FTP Server. ## Key Capabilities * Full-featured embedded FTP server with FTPS (FTP over TLS/SSL) support * Multi-client architecture with individual user profiles and directory isolation * Configurable permissions for send and receive directories per trading partner * Welcome banners, file filtering, and temporary file handling for reliable operations * Support for both active and passive modes with configurable port ranges ## Overview The Arc FTP Server is primarily configured on the [Profiles](../getting-started/administration/profiles) page. Then, individual FTP Server connectors provide a unique client profile to authenticate to the FTP Server. The client profile consists of a username (which matches the FTP Server connector's **Connector Id**), a password, a set of send and receive directories, and permissions to read and/or write files in these directories. FTP clients are not given permissions to the root of the FTP Server, which means that FTP clients should always *cd* into the send (to download) and receive (to upload) directories after connecting. ### Video Resources Watch this short video for an overview of how to configure an FTP Server. The video uses an SFTP Server in the example, but the principles are the same for FTP. [https://www.youtube.com/embed/R1OrVVdqySU](https://www.youtube.com/embed/R1OrVVdqySU) ## Profile Configuration The FTP Server Profile must be configured before connections can be established with individual FTP Server connectors. Click **Profiles** on the navbar, then click the **FTP Server** tab. ### Server Configuration *Server implementation settings.* | Setting | Description | | - | - | | **Port** | The port on which the FTP Server listens for incoming connections. | | **FTP Over TLS** | How TLS/SSL should be negotiated when clients connect to the server. Choose **Explicit** to establish a plain text connection where TLS/SSL is then started with an explicit command. Choose **Implicit** to immediately negotiate TLS/SSL without establishing a plain text connection. | | **Server TLS Certificate** | The TLS certificate that identifies the server. Only required if the server is operating as an FTPS server (in other words, if TLS/SSL is enabled). | | **Certificate Password** | The password required to access the **Server TLS Certificate**. | | **Welcome Message** | The banner presented to FTP clients when they connect to the server. | | **Root Directory** | The root directory for the server. Subfolders are created in the root for individual client profiles (in other words, for each configured FTP Server connector). Each client profile includes a *Send* folder, where clients can download files from the server, and a *Receive* folder, where clients can upload files to the server. | | **Allowed Files Filter** | A glob pattern that determines which files are accepted by the FTP server. You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Separate multiple file types by commas (for example, \*.x12,\*.edi). | ### Other Settings *Settings that do not fall into the previous category.* | Setting | Description | | - | - | | **Active Mode** | Check this to enable Active mode on the remote FTP server. Keep in mind that Passive mode is less likely to result in firewall interference with the connection. | | **Inactivity Timeout** | The length of time (in seconds) that must pass without activity for a user to time out. | | **Passive Port Range** | When using Passive mode, the application uses any available port to listen for incoming data connections. You can override this by setting this field to a port range in `start-end` format. For example: `1024-` refers to ports higher than 1024 (inclusive) and `1024-2048` refers to ports between 1024 and 2048 (inclusive). | | **Passive Address** | When using Passive mode, the application returns the IP address for the local interface. You can override this by setting this field to a specific public IP address. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Rotate Interval** The number of days to wait before creating a new log file.
**Log Delete Interval** The number of days to wait before deleting old log files.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Settings Once you configure the FTP Server profile settings, create and configure an individual FTP Server connector for each trading partner on the Flows page. ### Settings Tab #### Configuration
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Server Status *The current state of the connector's underlying service. The following statuses are available:*
Status Description
**Active** The service is running.
**Inactive** The service is configured but is not currently running.
**Not Configured** The underlying profile or service has not been set up.
Depending on whether you have already created the FTP profile, click the **Configure FTP Profile** or **View FTP Profile** links to open the profile in a new tab. #### User Configuration *Credentials for authenticating to the local FTP server.* | Setting | Description | | - | - | | **User** | The username credential for logging in to the local FTP server. This value is always the same as the **Connector Id**. | | **Password** | The password credential associated with the **User**. | #### Permissions *Settings related to the read and write permissions the configured client has for the Send and Receive folders.* | Setting | Description | | - | - | | **Send Directory Permissions** | Use the checkboxes to set read and write permissions for the *Send* directory. This directory is where files are downloaded. | | **Receive Directory Permissions** | Use the checkboxes to set read and write permissions for the *Receive* directory. This directory is where files are uploaded. | ### Advanced Tab #### Local Folders *Settings related to the folders where clients upload and download files.* | Setting | Description | | - | - | | **Input Folder (Send)** | Files in the Send folder are available to be downloaded by clients. | | **Output Folder (Receive)** | Files uploaded by the client are placed in the Receive folder. Files remain in the Receive folder or are passed along to the next connector in the flow. | #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Home Directory** | The configured server path where connected users start after logging in. For the valid locations and how the path is validated, see [Home Directory](#home-directory). | | **Allowed Files Filter** | A glob pattern that determines which files are accepted by the FTP server. Overrides the **Allowed Files Filter** option on the FTP [Profile](../getting-started/administration/profiles) page. | | **Remove File After Send** | Whether files in the *Send* folder should be removed after the client receives them. | | **Temp Receive Extensions** | Files with a matching extension are not recorded in the Receive table and do not fire the **After Receive** event until they are renamed. Supply a comma-delimited list of file extensions. | | **Timeout** | The length of time (in seconds) the server waits for a connection response before throwing a timeout error. | | **Save Subfolder** | Check this to add a SubFolder header to received messages. This header represents the path relative to the Local Folders. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Each FTP Server connector represents a single trading partner's connection parameters. The trading partner should connect to the FTP server using the server settings (such as port and TLS mode) from the [Profile](../getting-started/administration/profiles) page and the [user configuration](#user-configuration) settings in the dedicated FTP Server connector (User and Password). Each trading partner has separate *Send* and *Receive* directories that are subfolders of the root. The partner should download files from the Send folder and upload files to the Receive folder. The client is not permitted to upload or download files from the root. ## Home Directory The **Home Directory** setting controls which folder a client lands in when it logs in to an FTP Server connector. Configure it in the connector's **Advanced** tab, under **Other Settings**. Setting a home directory is useful when you migrate from other FTP software that let you configure a landing folder, or for automated systems that upload or download at whatever directory they land in without issuing navigation (`cd`) commands. The field defaults to empty, but it also accepts a server path: * **Empty (default):** The client lands at the virtual root (`/`). * **Set to a path:** On successful login, the client's initial working directory is the specified path. A client that connects and immediately uploads, downloads, or lists files without changing directories operates inside the home directory. The home directory must be a server path (starting with `/`) that resolves against the **Root Directory** defined on the connector's [Profile](#profile-configuration), and it must match a configured server location; you cannot set it to an arbitrary path that only exists on disk. For the FTP Server connector, the valid locations are: * `/` (the virtual root): this is the default when the field is empty, so setting it explicitly has no additional effect * `/Send` * `/Receive` The field normalizes your input, so `Receive`, `/Receive`, `\Receive`, and `/Receive/` are all treated as the same path. The value is validated when you save the connector: a path that does not match a configured server location is rejected with an error that names the invalid value. > [!NOTE] > > The home directory only sets the initial landing directory at login. It is not a chroot or jail and does not restrict which folders a client can navigate to afterward; folder access is still governed by the [permission settings](#permissions). ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Common Errors ### Error: Could not bind server socket: Permission denied. **Cause** This error can appear when you attempt to connect to an FTP server and the process hosting Arc does not have sufficient privileges to establish a listener on the specified port. > [!NOTE] > > In some cases (such as Linux environments and hosted instances running in an Amazon AMI), you cannot use ports below 1024. **Resolution** Choose a different port, or change the identity of the process hosting Arc to one with permissions to bind to the port. The Amazon AMI-hosted version of Arc uses the Ubuntu operating system, so CData recommends that you use an *Uncomplicated Firewall* (UFW) to manage port permission issues. For example, setting up FTP Server to run on port 8021 in Arc and using UFW to forward port 21 to 8021 at the OS level looks like this: ``` ufw allow 21/tcp ufw allow 8021/tcp echo " *nat :PREROUTING ACCEPT [0:0] -A PREROUTING -p tcp --dport 21 -j REDIRECT --to-port 8021 COMMIT" >> /etc/ufw/before.rules ``` If your environment uses a different Linux operating system, CData recommends that you bind to a port above the restricted range (for example, 8021 for FTP traffic) and use `iptables` to route incoming requests on the desired port to the allowed port: `iptables -t nat -I PREROUTING -p tcp --dport 21 -j REDIRECT --to-port 8021` # GISB Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/gisb Configuration and usage guide for the CData Arc GISB connector, which supports sending and receiving messages using the Gas Industry Standards Board (GISB) protocol. GISB connectors support sending and receiving messages using the Gas Industry Standards Board (GISB) protocol. ## Key Capabilities * GISB protocol support for energy sector B2B communications * Digital signature and encryption capabilities with key management * Receipt processing and acknowledgment handling for reliable delivery * Support for multiple GISB versions with configurable protocol options ## Overview A GISB connection is configured in two places. Configure the GISB [Profiles](../getting-started/administration/profiles) page with a local GISB identifier, personal key, and other information that is global across all GISB connections. Then configure individual GISB connectors with connection settings specific to a single trading partner. When a GISB connector processes an input file, it is packaged and sent to the specified trading partner. When Arc receives a file over GISB, it attempts to route the file to a specific GISB connector. The application uses the GISB identifiers in the message to determine which GISB connector should receive the file. When a file is routed to a GISB connector, that file is placed on the connector's **Transactions** tab, or is passed along to the next connector in the flow. ## Profile Configuration The GISB profile must be configured before connections can be established with individual GISB connectors. Click **Profiles > GISB** on the navbar. ### GISB Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **GISB Identifier** | Your GISB identifier. Messages sent by Arc include this value as the `GISB-From` header. Incoming messages must have this value as the `GISB-To` header to be successfully received. GISB Identifiers are case-sensitive. | #### Personal Key *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Secret Key** | The key that is used to decrypt incoming messages and sign outgoing messages. | | **Passphrase** | The password required to access the local secret key. | #### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Receiving URL** | The URL at which Arc listens for incoming GISB messages. Share this URL with all your trading partners. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Trading Partner Info *Settings for identifying and connecting to a specific trading partner.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **GISB Identifier** | The identifier specific to the target trading partner. This value is included in outgoing messages, and it is also used to route incoming messages to the appropriate GISB connector. | | **Partner URL** | The trading partner's public endpoint where outgoing messages should be sent. | | **GISB Version** | The protocol version of the GISB client. | #### Connection Info *Settings related to connection parameters for the specified trading partner.* | Setting | Description | | - | - | | **Send Message Security** | Whether to sign and/or encrypt outgoing messages. CData strongly recommends that you use signatures and encryption. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming messages. An error is thrown if a received message does not have a required security parameter. | | **Client Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | #### Receipts *Settings related to sending and requesting message receipts.* | Setting | Description | | - | - | | **Security** | Whether receipts should include a signature block verifying the message integrity and identity of the recipient. | #### Trading Partner Keys/Certificates *Settings related to the public certificates provided by the trading partner.* | Setting | Description | | - | - | | **Recipient Key Certificate** | The public key used for encryption and signature verification. Use the **Import Key** button, then browse to the key to import. | | **TLS Server Certificate** | The public certificate used to verify the identity of a TLS/SSL server. This is only necessary if the partner's GISB system requires HTTPS instead of HTTP. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server's identity. | ### Advanced Tab #### Alternate Local Profile *Settings that override the GISB configuration on the Profiles page for this specific GISB connector. Setting an alternate local profile allows the use of different local keys and identifiers for individual trading partners.* | Setting | Description | | - | - | | **Local GISB Identifier** | Your GISB identifier. | | **Secret Key** | The key that is used to decrypt incoming messages and sign outgoing messages. | | **Passphrase** | The password required to access the local secret key. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.*
SettingDescription
**Private Certificate** The private certificate presented during TLS client authentication.
**Certificate Password** The password required to access the TLS client certificate.
#### HTTP Authentication *Settings related to HTTP client authentication.* | Setting | Description | | - | - | | **HTTP Authentication** | Whether to use client HTTP authentication. | | **HTTP Authentication Type** | Whether to provide HTTP authentication credentials in an encrypted format (Digest) or in plain text (Basic). Only use Basic authentication if the connection is an HTTPS connection (instead of HTTP). | | **User** | The user credential for HTTP client authentication. | | **Password** | The password credential for HTTP client authentication. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Compression Method** | The algorithm to use when compressing outgoing payloads. | | **EDI Type** | The type of EDI message to use. The default is X12. | | **Encrypting Algorithm** | The algorithm to use when encrypting outgoing messages. | | **Fold Headers** | Check this to have the connector fold the headers if the headers are over a certain length. | | **Packet Type** | Check this to have the connector read the features from the **Recipient Key Certificate** to determine if the older packet type is required. If the key requires the older packet type, it is used. | | **Sign and Encrypt** | Check this to have the connector perform signing and encryption in the same step. | | **Signing Algorithm** | The algorithm to use when signing outgoing messages. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Temp File** | Check this to have the connector write data to be processed to a temp file, then move the temp file into processing after it has been fully written. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as GISB messages. | | **Retry Interval** | The amount of time before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If this is set to 0, the connector retries the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Partner Setup Trading partners must share the connection details that are required when configuring a new GISB connector. At a minimum, these details must include: * GISB Identifier * Partner URL * Recipient Key Certificate ## Send and Receive Files Once the GISB profile and partner-specific GISB connectors are configured, files can be securely sent and received. ### Send Files In a GISB connector, the **Transactions** tab displays the files to be sent to the target trading partner. If **Send** is enabled on the [Automation](#automation-tab) tab, files that reach this tab are automatically packaged and sent. Access the log files for all transmissions by expanding the row associated with the transmitted file. The **Create Test Files** button lets you generate a simple series of test files to send to the trading partner. ### Receive Files In a GISB connector, the **Transactions** tab displays the files that have been received by the application and routed to the connector (based on the GISB identifiers present in the incoming GISB message). Expand each file row to display a list of available logs for the transmission. These files are available on the connector **Transactions** tab. If the connector is connected to other connectors in the flow, files are automatically moved from the **Transactions** tab of the GISB connector to the **Transactions** tab of the next connector in the flow. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Google Drive Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/google-drive Configuration and usage guide for the CData Arc Google Drive connector, which uploads to and downloads from the Google Drive cloud storage service. The Google Drive connector uploads to and downloads from the Google Drive cloud storage service. ## Key Capabilities * OAuth-based Google Drive integration with automatic token management * Support for Google Workspace document export in multiple formats (Docs, Sheets, Slides) * Configurable send and receive scopes for different Google Drive areas * Advanced file filtering and recursive directory operations * Optional caching to ensure only new or updated files are downloaded ## Overview Each Google Drive connector connects to a specific Google Drive storage account. Within the remote storage, you set a single folder as the upload target, but you can use a comma-separated list to supply multiple download targets. Files that reach the Google Drive connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specific file names and/or file extensions. Account authentication uses OAuth, so you must enter account credentials in an external site portal the first time the connector connects. The external portal generates an OAuth token that the connector uses to connect, then the connector handles token refresh automatically. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization *Settings related to authorizing access to the remote storage.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Client Id** | The client Id credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. | | **Client Secret** | The client secret credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. | > [!NOTE] > > Google might require an OAuth callback/redirect URL when generating OAuth credentials. The Arc callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: `src/oauthCallback.rst`. For example, if Arc is hosted on 'mydomain.com' on port 8001, specify the following URL as the callback/redirect URL: `http://mydomain.com:8001/src/oauthCallback.rst`. #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Send Scope** | Use the dropdown to choose where to upload files. | | **Remote Folder** | Files processed by the connector are uploaded to the folder specified here. | | **Overwrite** | Whether files that already exist in the remote folder should be overwritten or skipped. | > [!NOTE] > > When you upload to a shared drive where the authenticated account is a *contributor* (not an *owner* or *organizer*), the **Overwrite remote files** feature is not supported. This is a limitation imposed by Google's Drive API permission model. #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Receive Scope** | Use the dropdown to choose where to download files from. | | **Remote Folder** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | **File Mask** | A glob pattern that determines which files in the **Remote Folder** should be downloaded (for example, \*.txt). If you need to specify multiple file masks, use the **Receive Filter** in the Advanced Settings instead. | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **Delete Mode** | Files can exist in multiple folders in Google Drive. Deleting a file can remove it from all folders (moving the file to trash) or disassociate it with a single folder. Select *Delete Resource* to remove the file from all folders, and select *Remove Parent* to remove the file from the parent folder set in **Remote Folder**. | | **Docs Export Format** | Select the MIME type to export to when downloading Google Docs. | | **Sheets Export Format** | Select the MIME type to export to when downloading Google Sheets. | | **Slides Export Format** | Select the MIME type to export to when downloading Google Slides. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection OAuth credentials are required to establish a connection with the Google Drive connector. Acquire the OAuth credentials from Google via the Google Console, then specify them in the following connector fields on the [Settings tab](#settings-tab): * OAuth Client ID * OAuth Client Secret Google might require an OAuth callback/redirect URL when generating OAuth credentials. The Arc callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: `src/oauthCallback.rst`. For example, if Arc is hosted on 'mydomain.com' on port 8001, specify the following URL as the callback/redirect URL: `http://mydomain.com:8001/src/oauthCallback.rst`. Once authentication is successfully completed with Google, the Google Drive connector handles refreshing the OAuth tokens to ensure that authentication persists over time. ## Uploading and Downloading ### Upload To upload files, set the **Remote Folder** field in the [Send](#send) portion of the **Settings** page to the folder where files should be uploaded. Each Google Drive connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the remote folder should be overwritten or skipped. > [!NOTE] > > When you upload to a shared drive where the authenticated account is a *contributor* (not an *owner* or *organizer*), the **Overwrite remote files** feature is not supported. This is a limitation imposed by Google's Drive API permission model. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Download To download files, set the **Remote Folder** field in the [Receive](#receive) portion of the **Settings** page to the folder where files should be downloaded. Use a comma-delimited list to specify multiple folders. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the [Advanced](#advanced-tab) tab. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. After configuration is complete, files are downloaded according to the **Receive** [Automation](#automation-settings) settings, or you can manually click the **Receive** button on the **Transactions** tab. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Google Storage Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/google-storage Configuration and usage guide for the CData Arc Google Storage connector, which uploads to and downloads from the Google Cloud Storage service. The Google Storage connector uploads to and downloads from the Google Storage cloud storage service. ## Key Capabilities * Google Cloud Storage integration with OAuth authentication and automatic token refresh * Bucket-based file organization with bidirectional file transfers * Glob pattern filtering with optional file deletion after download and overwrite protection * Optional caching to ensure only new or updated files are downloaded ## Overview Each Google Storage connector connects to a specific Google Storage account. Within the remote storage, you set a single bucket as the upload and/or download target. You can specify upload and download paths in the bucket, and apply filters to only send or receive specified file names and/or file extensions. Files that reach the Google Storage connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. Account authentication uses OAuth, and you must acquire OAuth credentials before connecting with the Google Storage connector. Once authentication is successfully completed with Google, the Google Storage connector handles refreshing the OAuth tokens to ensure that authentication persists over time. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization *Settings related to authorizing access to the remote storage.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Client Id** | The client Id credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. | | **Client Secret** | The client secret credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. | > [!NOTE] > > Google might require an OAuth callback/redirect URL when generating OAuth credentials. The Arc callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: `src/oauthCallback.rst`. For example, if Arc is hosted on 'mydomain.com' on port 8001, specify the following URL as the callback/redirect URL: `http://mydomain.com:8001/src/oauthCallback.rst`. #### Host Configuration *Settings related to the target storage host.* | Setting | Description | | - | - | | **Bucket Name** | The Google Storage bucket to use for uploading and/or downloading files. | #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | Files processed by the connector are uploaded to the folder specified here. | | **Overwrite** | Whether files that already exist in the remote folder should be overwritten or skipped. | #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **Use Virtual Hosting** | By default (*Auto*), the connector tries to determine which request style to use based on the bucket name. Set to *True* to force the buckets to be referenced using a hosted-style request: `http://yourbucket.storage.googleapis.com/yourobject`. Set to *False* to use a path-style request: `http://storage.googleapis.com/yourbucket/yourobject`. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection OAuth credentials are required to establish a connection with the Google Storage connector. Acquire the OAuth credentials from Google via the Google Console, then specify them in the following connector fields on the [Settings tab](#settings-tab): * OAuth Client ID * OAuth Client Secret Google might require an OAuth callback/redirect URL when generating OAuth credentials. The Arc callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: `src/oauthCallback.rst`. For example, if Arc is hosted on 'mydomain.com' on port 8001, specify the following URL as the callback/redirect URL: `http://mydomain.com:8001/src/oauthCallback.rst`. Once authentication is successfully completed with Google, the Google Storage connector handles refreshing the OAuth tokens to ensure that authentication persists over time. ## Uploading and Downloading ### Upload To upload files, set the **Bucket Name** field in the [Host Configuration](#host-configuration) portion of the **Settings** page to the target bucket, then set the **Remote Folder** field in the [Send](#send) portion of the **Settings** page to the folder where files should be uploaded. Each Google Storage connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the remote folder should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Download To download files, set the **Remote Folder** field in the [Receive](#receive) portion of the **Settings** page to the folder where files should be downloaded. Use a comma-delimited list to specify multiple folders. You can set the **Receive Filter** field on the [Advanced tab](#advanced-tab) to a glob filter to specify which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use a comma-separated list to specify multiple filters. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. After configuration is complete, files are downloaded according to the **Receive** [Automation](#automation-settings) settings, or you can manually click the **Receive** button on the **Transactions** tab. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# HL7 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/hl7 Configuration and usage guide for the CData Arc HL7 connector. Health Level-7 is a set of standards for the transfer of clinical data used by hospitals and other healthcare providers. HL7 connectors support generating HL7 documents from XML and converting HL7 documents into XML. ## Key Capabilities * Healthcare data exchange with full HL7 message standard support * MSH header validation and generation for compliant message structure * Original mode acknowledgment processing for reliable healthcare communications * Support for multiple HL7 versions with strict schema validation options ## Overview When receiving HL7 documents, HL7 connectors validate interchange headers and convert the HL7 document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The HL7 connector automatically reads the input file to determine the appropriate schema, then parses the document according to this schema. When generating HL7 documents, HL7 connectors convert XML into HL7 document syntax and apply the appropriate interchange headers. This is useful as the final step for creating an HL7 document, after the XML data has been fetched and transformed elsewhere in the flow. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert HL7 documents into XML, or generate HL7 documents from XML. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner Id (MSH3.1)** | In conjunction with the other 3.X elements, identifies the application sending the HL7 document. Only applicable when **Dynamically process partners** is unchecked. | | **Personal Id (MSH5.1)** | In conjunction with the other 5.X elements, identifies the application receiving the HL7 document. Only applicable when **Dynamically process partners** is unchecked. | #### MSH Settings *Settings related to the message headers. When converting HL7 documents into XML, the document headers are validated against these settings; when generating HL7 documents from XML, these settings are used to generate document headers.* | Setting | Description | | - | - | | **Partner Application Universal Id (MSH3.2)** | In conjunction with the other 3.X elements, identifies the application sending the HL7 document. | | **Partner Application Universal Id Type (MSH3.3)** | In conjunction with the other 3.X elements, identifies the application sending the HL7 document. | | **Partner Facility Namespace Id (MSH4.1)** | In conjunction with the other 4.X elements, identifies the facility sending the HL7 document. | | **Partner Facility Universal Id (MSH4.2)** | In conjunction with the other 4.X elements, identifies the facility sending the HL7 document. | | **Partner Facility Universal Id Type (MSH4.3)** | In conjunction with the other 4.X elements, identifies the facility sending the HL7 document. | | **Personal Application Universal Id (MSH5.2)** | In conjunction with the other 5.X elements, identifies the application receiving the HL7 document. | | **Personal Application Universal Id Type (MSH5.3)** | In conjunction with the other 5.X elements, identifies the application receiving the HL7 document. | | **Personal Facility Namespace Id (MSH6.1)** | In conjunction with the other 6.X elements, identifies the facility receiving the HL7 document. | | **Personal Facility Universal Id (MSH6.2)** | In conjunction with the other 6.X elements, identifies the facility receiving the HL7 document. | | **Personal Facility Universal Id Type (MSH6.3)** | In conjunction with the other 6.X elements, identifies the facility receiving the HL7 document. | The following options are only visible when the **Translation Type** is XML to HL7. | Setting | Description | | - | - | | **Security (MSH8)** | Used to implement security features. | | **Processing Id (MSH11.1)** | Defines whether the document is part of a production, training, or debugging system. | | **Processing Mode (MSH11.2)** | Defines whether the document is part of an archival process or an initial load. Options are: not present-current processing, archive, restore from archive, or initial load. | | **Version Id (MSH12)** | This field is matched by the receiving system to its own version to ensure the document is interpreted correctly. | | **Sequence Number (MSH13)** | A non-null value in this field indicates that the sequence number protocol is in use. | | **Continuation Pointer (MSH14)** | Used to define continuations in application-specific ways. | | **Accept Acknowledgment Type (MSH15)** | Identifies the conditions under which accept acknowledgments are required to be returned in response to the document. Required for enhanced acknowledgment mode. Options are: always, never, error or reject conditions only, or successful completion only. | | **Application Acknowledgment Type (MSH16)** | Contains the conditions under which application acknowledgements are required to be returned in response to the document. Required for enhanced acknowledgment mode. Options are: always, never, error or reject conditions only, or successful completion only. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **Acknowledgment Type** | Whether or not acknowledgments should be generated and requested. See [Acknowledgments](#acknowledgements) for more information. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Repetition Char** | The character that indicates repetition of element values. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | Whether multiple transactions should be grouped together in a single output file. When this is not checked, the connector creates a separate output file for each transaction. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is HL7 to XML. | | **Expand Qualifier Values** | When translating HL7 into XML, whether elements containing an HL7 qualifier should include child elements to express the qualifier code and value. Only applicable when the **Translation Type** is HL7 to XML. For example: ``   `ST`   `Ship To` `` | | **Generate Description As** | When translating HL7 into XML, descriptions of the HL7 segments and elements can be provided as context for the HL7 data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. | | **Transmission Control** | Whether the connector should wait for a transmission to be acknowledged before beginning the next transmission. Only applicable when the **Translation Type** is XML to HL7. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Validate Identifiers** | When checked, the connector ensures that the identifiers in the translated document match the identifiers specified in the connector configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_hl7_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_hl7_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## XML to HL7 When generating HL7 files, set the **Translation Type** to *XML to HL7*. The application then reads the header information from the settings configured on the [Settings tab](#settings-tab). Once you have configured the connector with the settings from your trading partner agreement, click the **Transactions** tab and select **More > Create Test Files**. The application creates a set of test XML files which provide example data of what the XML should look like for the connector to generate an HL7 document. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting HL7 document. ## HL7 to XML When generating XML, set the **Translation Type** to *HL7 to XML*. Then complete the trading partner agreement settings so the application can validate the information in the HL7 documents during processing. XML files are generated into the Output folder. To create test HL7 files for several HL7 documents, click the **Transactions** tab and select **More > Create Test Files**. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting XML. ## Use With the XML Map Connector In accordance with Arc convention, the HL7 connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound HL7 documents, or as the following step in the flow when receiving inbound HL7 documents. The interaction between the XML Map connector and the HL7 connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the HL7 connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the HL7 connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the HL7 document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## Acknowledgements Acknowledgements can be automatically generated and processed by the application. #### Original Mode Acknowledgements This is an indication that an interchange has taken place between the two parties (although not necessarily that any individual message has been exchanged). It is automatically generated in response to a request in the interchange. ### Message Originator #### Requesting Acknowledgements HL7 connectors in XML to HL7 mode can be configured so that original mode acknowledgements are requested for an interchange. Select *Original Mode* in the **Acknowledgement Type** field on the [Settings tab](#settings-tab) to direct the connector to maintain a Pending ACK status for the transmission until the appropriate acknowledgements have been returned and processed. ![Receive ACK](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-receive.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=e753246ca35494ac86549872227df731) The diagram above shows a complete message exchange from the standpoint of the message originator. An HL7 connector in HL7 to XML mode generates the document to be exchanged **(1)** and holds it in a Pending ACK state. The trading partner processes the transmission in their business logic and creates acknowledgements in accordance with the configured exchange parameters and the bidirectional agreement between parties **(2)**. When the acknowledgements are returned, they are parsed with an HL7 to XML HL7 connector and routed to the original XML to HL7 connector to resolve the status of the transfer **(3)**. > [!NOTE] > > The **Transmission Control** setting in the [Advanced Settings](#advanced-settings) section of the **Advanced** tab controls the flow of documents as it relates to their acknowledgements. When this is checked, the connector waits for an acknowledgement of the current message before processing the next message. #### Processing Acknowledgements When HL7 acknowledgements are received at an HL7 connector in HL7 to XML mode, the connector can be configured to automatically route any detected acknowledgements to the XML to HL7 connector that originated the transmission. Routing ACKs between HL7 connectors is configured visually on the Flows canvas by dragging the gray dot at the bottom of the HL7 connector in XML to HL7 mode onto the HL7 connector that is in HL7 to XML mode. The receiving XML to HL7 connector then pairs the acknowledgements to the original message and resolves the matching message. ### Message Recipient #### Generating Acknowledgements When an HL7 connector in HL7 to XML mode is processing received messages from the originator and generating XML, you can tell the connector to automatically generate original mode acknowledgements by selecting *Original Mode* in the **Acknowledgement Type** field on the [Settings tab](#settings-tab). You must route these acknowledgements to an HL7 connector in XML to HL7 mode to generate the HL7 acknowledgement. To route the ACK appropriately, drag the gray dot at the bottom of the HL7 connector in HL7 to XML mode onto the HL7 connector that is in XML to HL7 mode. ![Generate ACKs](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/edifact-ack-create.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=3d2adfeb1f049885a0cf400ccc9b6cb5) As shown in the diagram above, when a trading partner sends a message where an acknowledgement is expected **(1)**, the HL7 to XML HL7 connector configured for that partner automatically generates an acknowledgement **(2)** as an XML file containing the transactional information relevant for that file. This XML acknowledgement must be routed back to an HL7 connector in XML to HL7 mode **(3)** so that all of the HL7 party agreement settings used when generating the HL7 file are sent back to the trading partner. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# HL7 MLLP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/hl7mllp Configuration and usage guide for the CData Arc HL7 MLLP connector. Health Level-7 is a set of standards for the transfer of clinical data used by hospitals and other healthcare providers. HL7 MLLP connectors support generating and parsing HL7 documents in conjunction with MLLP as the transport protocol. ## Key Capabilities * Healthcare data exchange using HL7 over MLLP (Minimal Lower Layer Protocol) transport * Embedded MLLP server functionality with SSL/TLS encryption support * Bidirectional HL7 document translation between HL7 and XML formats * MSH header validation with configurable application and facility identifiers * Automatic acknowledgment handling with retry and resend capabilities for reliable delivery ## Overview HL7 MLLP connectors provide two sets of functionality: * File transfer over the MLLP transport protocol * EDI document parsing and generation according to the HL7 EDI standard An HL7 MLLP connection is configured in two places. Configure the HL7 MLLP [Profiles](../getting-started/administration/profiles) page with the settings that identify Arc as an HL7 and MLLP entity. Then configure individual HL7 MLLP connectors with the HL7 and MLLP details for each trading partner. When an HL7 MLLP connector receives an inbound message from a trading partner, it verifies the HL7 identification details to confirm that the document was sent from and to the expected parties. Then it translates the document into XML for further processing in a Arc flow. When an HL7 MLLP connector sends files to a trading partner: * XML files are translated to HL7 before being sent * HL7 documents are sent as-is CData strongly recommends that you use the HL7 MLLP connector in conjunction with the [XML Map connector](./xml-map/xml-map) to ensure that input and output XML is formatted correctly. See [Use with the XML Map Connector](#use-with-the-xml-map-connector) for more information. ## Profile Configuration The HL7 MLLP Profile must be configured before connections can be established with individual HL7 MLLP connectors. Click **Profiles** on the navbar, then click the **HL7 MLLP** tab. ### HL7 MLLP Profile Tab #### Server Configuration *Settings for hosting a local MLLP server.* | Setting | Description | | - | - | | **Port** | The port on which to listen for incoming MLLP connections. | | **Use SSL/TLS** | Whether to require incoming connections to use SSL/TLS encryption for transport security. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. If the target server uses TLS/SSL, set this field to the certificate that identifies the server. If the server's certificate is not available, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to implicitly trust the server's identity. (This field only appears if you check **Use SSL/TLS**.) | #### HL7 Personal Ids | Setting | Description | | - | - | | **Application Namespace Id (MSH3.1)** | The application Id that identifies the local namespace for the HL7 protocol exchange. | | **Application Universal Id (MSH3.2)** | The local application universal Id for the HL7 protocol exchange. | | **Application Universal Id Type (MSH3.3)** | A type qualifier to provide context to the local application universal Id. | | **Facility Namespace Id (MSH4.1)** | The facility Id that identifies the local namespace for the HL7 protocol exchange. | | **Facility Universal Id (MSH4.2)** | The local facility universal Id for the HL7 protocol exchange. | | **Facility Universal Id Type (MSH4.3)** | A type qualifier to provide context to the local facility universal Id. | #### Other Settings *Settings that rarely need to be modified.* | Setting | Description | | - | - | | **Inactivity Timeout** | The amount of time (in seconds) that the server allows clients to remain inactive before the connection is terminated. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Rotate Interval** The number of days to wait before creating a new log file.
**Log Delete Interval** The number of days to wait before deleting old log files.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration Once you configure the global HL7 MLLP profile settings, create and configure individual HL7 MLLP connectors for each trading partner on the Flows page. ### Settings Tab #### Configuration
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Connector Settings ##### Server Status *The current state of the connector's underlying service. The following statuses are available:*
Status Description
**Active** The service is running.
**Inactive** The service is configured but is not currently running.
**Not Configured** The underlying profile or service has not been set up.
Depending on whether you have already created the HL7 MLLP profile, click the **Configure HL7 MLLP Profile** or **View HL7 MLLP Profile** links to open the profile in a new tab. *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Remote Host** | The hostname or IP address of your remote host. | | **Port** | The port on which to listen for incoming MLLP connections. | | **Version** | The version of MLLP being used. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Use SSL/TLS** | Whether to require incoming connections to use SSL/TLS encryption for transport security. | #### HL7 Configuration *Settings related to the message headers. When converting HL7 documents into XML, the document headers are validated against these settings; when generating HL7 documents from XML, these settings are used to generate document headers.* | Setting | Description | | - | - | | **Application Namespace Id (MSH5.1)** | In conjunction with the other 5.X elements, identifies the application on the remote side of the HL7 exchange. | | **Application Universal Id (MSH5.2)** | In conjunction with the other 5.X elements, identifies the application on the remote side of the HL7 exchange. | | **Application Universal Id Type (MSH5.3)** | In conjunction with the other 5.X elements, identifies the application on the remote side of the HL7 exchange. | | **Facility Namespace Id (MSH6.1)** | In conjunction with the other 6.X elements, identifies the facility on the remote side of the HL7 exchange. | | **Facility Universal Id (MSH6.2)** | In conjunction with the other 6.X elements, identifies the facility on the remote side of the HL7 exchange. | | **Facility Universal Id Type (MSH6.3)** | In conjunction with the other 6.X elements, identifies the facility on the remote side of the HL7 exchange. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **Acknowledgment Type** | Whether or not acknowledgments should be generated and requested. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Repetition Char** | The character that indicates repetition of element values. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Generate Description As** | When translating HL7 into XML, descriptions of the HL7 segments and elements can be provided as context for the HL7 data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Validate Identifiers** | When checked, the connector ensures that the identifiers in the translated document match the identifiers specified in the connector configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as HL7 or XML messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connector retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous MDN receipt is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Generating HL7 Documents When generating HL7 files from XML, the connector applies transaction headers based on the [HL7 Personal Ids](#hl7-personal-ids) on the HL7 MLLP Profile tab, and the [HL7 Configuration](#hl7-configuration) in the connector. The remaining data comes from XML input files. See [Use with the XML Map Connector](#use-with-the-xml-map-connector) for more information on generating the appropriate XML input files. Files that arrive in the **Transactions** tab for the connector are packaged into HL7 documents and sent out over MLLP. Files can arrive in the **Transactions** tab either by dropping the file directly into the folder path on disk, or by connecting another Arc connector to the HL7 MLLP connector in the flow. ## HL7 to XML When converting HL7 documents into XML, the application verifies the interchange headings from the inbound document against the [HL7 Personal Ids](#hl7-personal-ids) on the HL7 MLLP Profile tab, and the [HL7 Configuration](#hl7-configuration) in the connector. The XML generated from the HL7 document is placed in the **Transactions** tab to be passed along in the flow for further processing. See [Use with the XML Map Connector](#use-with-the-xml-map-connector) for more information on processing the translated XML. ## Use With the XML Map Connector In accordance with Arc convention, the HL7 MLLP connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound HL7 MLLP documents, or as the following step in the flow when receiving inbound HL7 MLLP documents. The interaction between the XML Map connector and the HL7 MLLP connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the HL7 MLLP connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the HL7 MLLP connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the HL7 document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# IATA PADIS Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/iatapadis Configuration and usage guide for the CData Arc IATA PADIS connector. International Air Transport Association (IATA) Passenger and Airport Data Interchange Standards (PADIS) connectors support generating PADIS documents from XML and converting PADIS documents into XML. ## Key Capabilities * IATA PADIS support with bidirectional IATA PADIS/XML translation * Automatic EDIFACT-subset schema detection and validation for aviation industry messaging * CONTRL acknowledgment generation/processing with interchange header management for routing ## Overview When receiving PADIS documents, IATA PADIS connectors validate document interchange headers and convert the PADIS document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The IATA PADIS connector automatically reads the input file to determine the appropriate PADIS schema, then parses the document according to this schema. IATA PADIS is a subset of the EDIFACT standard, so many of the features and functionality of the IATA PADIS connector are shared with the more general [EDIFACT connector](./edifact). When generating PADIS documents, IATA PADIS connectors convert XML into PADIS document syntax and apply the appropriate interchange headers. This is useful as the final step for creating a PADIS document, after the XML data has been fetched and transformed elsewhere in the flow. > [!NOTE] > > Interchange header validation can be avoided by enabling the **Test Indicator** setting. An IATA PADIS connector can also automatically generate acknowledgments to incoming PADIS documents. For more information, see [IATA PADIS Acknowledgments](#iata-padis-acknowledgments). ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert PADIS documents into XML, or generate PADIS documents from XML. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner Identifier (UNB2.1)** | The unique Id identifying the sending party in the PADIS communication. When you are generating a PADIS document, this should be your identifier. Only applicable when **Dynamically process partners** is unchecked. | | **Recipient Identifier (UNB3.1)** | The unique Id identifying the receiving party in the PADIS communication. When you are generating a PADIS document, this should be your trading partner's identifier. Only applicable when **Dynamically process partners** is unchecked. | #### Interchange Settings *Settings related to the IATA PADIS interchange headers. When generating PADIS documents from XML, these settings are used to generate document headers. When parsing PADIS documents, these settings are used to validate the incoming document.* | Setting | Description | | - | - | | **Syntax Identifier (UNB1.1)** | Identifies the character set used in the PADIS document. | | **Syntax Version (UNB1.2)** | In combination with the **Syntax Identifier**, determines the syntax to be used in the PADIS document. Based on your selection here, other interchange setting options appear or disappear on the [Advanced tab](#advanced-tab). | | **Partner Code Qualifier (UNB2.2)** | The qualifier for the **Partner Identifier**, which provides context to the value (for example, an EAN location number). | | **Personal Code Qualifier (UNB3.2)** | The qualifier for the **Personal Identifier**, which provides context to the value (for example, an EAN location number). | | **Test Indicator (UNB11)** | Whether the interchange is in test mode or production mode. If you check this, interchange headers are *not* validated when documents are received. | | **Functional Group** | Check this to automatically add the sender and recipient identifiers to the [Functional Group Settings](#functional-group-settings) on the **Advanced** tab. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **Technical acknowledgment (CONTRL)** | Whether a technical CONTRL ACK should be returned (when receiving) and requested (when sending). A technical acknowledgment serves as a receipt of the interchange. | | **Functional acknowledgment (CONTRL)** | Whether a functional CONTRL ACK should be returned (when receiving) and requested (when sending). A functional acknowledgment serves as an indication of acceptance or rejection of the received interchange. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, etc.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Repetition Char** | The character that indicates repetition of element values. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Interchange Settings *Additional settings related to the IATA PADIS interchange headers. These options appear or disappear based on the **Syntax Version** specified on the Settings tab.* | Setting | Description | | - | - | | **Service Code List Directory Version Number (UNB1.3)** | Further specifies the syntax to be used in the PADIS document. Only applicable for PADIS syntax version 4. | | **Character Encoding (UNB1.4)** | Specifies how characters are encoded (such as ASCII or UTF-8). Only applicable for PADIS syntax version 4. | | **Address for Reverse Routing (UNB2.3)** | The optional address in the sender's system to which responding interchanges should be sent. Only applicable for PADIS syntax versions prior to version 4. | | **Partner Internal Identification (UNB2.3)** | An additional sender identifier to facilitate internal routing of response interchanges. Only applicable for PADIS syntax version 4. | | **Partner Internal Sub-Identification (UNB2.4)** | Further identifies the sender when sub-level identification is required. Only applicable for PADIS syntax version 4. | | **Routing Address (UNB3.3)** | The optional address in the recipient's system to which interchanges should be routed. Only applicable for PADIS syntax versions prior to version 4. | | **Personal Internal Identification (UNB3.3)** | An additional recipient identifier to facilitate internal routing of received interchanges. Only applicable for PADIS syntax version 4. | | **Personal Internal Sub-Identification (UNB3.4)** | Further identifies the recipient, for when sub-level identification is required. Only applicable for PADIS syntax version 4. | | **Partner Password (UNB6.1)** | Reference or password to gain access to the recipient's system. | | **Partner Password Qualifier (UNB6.2)** | The qualifier that provides context to the **Partner Password**, if applicable. | | **Application Reference ID (UNB7)** | Identifies the application group to which the messages in the interchange relate. | | **Processing Priority Code (UNB8)** | Code for requesting processing priority for the interchange. | | **Communication Agreement (UNB10)** | Defines the type of communication agreement controlling the interchange. | #### Functional Group Settings *Settings related to the functional group headers of PADIS documents. These optional identifiers can help group similar interchanges together or facilitate sub-addressing within an organization.* | Setting | Description | | - | - | | **Application Partner Identifier (UNG2.1)** | Identifies the application sending the document (such as a division, branch, or computer system). | | **Application Personal Identifier (UNG2.1)** | Identifies the application for which the document is intended. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An interchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the interchange. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is IATA to XML. | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing an EDI qualifier include child elements containing the qualifier code and value. For example: ``   `ST`   `Ship To` `` | | **Functional Acks** | By default, all functional acknowledgments (997, 999) are routed to the connector selected in the flow diagram, and XML translations are not received in the **Transactions** tab. Check this to have the translated acknowledgment also be included in the **Transactions** tab. This allows functional acknowledgments to be integrated into a destination source in addition to new PADIS documents. | | **Generate Description As** | When translating PADIS into XML, descriptions of the PADIS segments and elements can be provided as context for the PADIS data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects PADIS structures that have hierarchical relationships embedded in the PADIS data, and generates XML with these hierarchical relationships represented as parent-child relationships. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Track UNB2.1** | Whether to add UNB2.1 values as tracked headers to processed messages. | | **Track UNB3.1** | Whether to add UNB3.1 values as tracked headers to processed messages. | | **Track Transaction Types** | Whether to add transaction types as tracked headers to processed messages. | | **Validate Identifiers** | Check this to ensure that the identifiers in the translated document match the identifiers in the connector's configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_iatapadis_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_shared_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting Formats The following sections detail the process of converting PADIS documents to XML and vice versa. ### PADIS to XML Setting the **Translation Type** to *IATA to XML* instructs the connector to parse incoming PADIS documents into XML. The connector first reads all of the header information for the Interchange and Functional Group sections of the document and validates them against the configured connector settings (unless **Test Indicator** is enabled). The connector then parses out the specific PADIS schema used in the document and loads the schema from the `edifact_schemas` folder on disk (additional schema files can be [downloaded from our website](https://arc.cdata.com/schemas/) for free). Using the schema, the connector generates XML representing the structure of the document, populates the XML with the values from the document, and provides context to each value either as XML comments or as attributes in the XML elements (based on the value of **Generate Descriptions As**). To see this process with a set of test PADIS documents, navigate to the **Transactions** tab of an IATA PADIS connector (with **Translation Type** set to *IATA to XML*) and select **More > Create Test Files**. PADIS documents for an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting XML. Once PADIS documents are converted to XML, the data can be transformed and manipulated in many ways. Commonly, PADIS data needs to be stored in a database or other back-end application system. Since Arc uses XML to represent `inserts` into these back-end systems, storing the PADIS data becomes a matter of mapping one XML structure onto another. This is typically done with the visual designer-driven [XML Map Connector](./xml-map/xml-map). ### XML to PADIS Setting the **Translation Type** to *XML to IATA* instructs the connector to generate a PADIS document out of an XML representation of the document. After the connector has constructed the PADIS message out of the data parsed from the XML, it adds functional group and interchange headers according to the configured connector settings. To see this process with a set of test XML files, navigate to the **Transactions** tab of an IATA PADIS connector (with **Translation Type** set to *XML to IATA*) and select **More > Create Test Files**. XML files representing an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting PADIS document. ## Use With the XML Map Connector In accordance with Arc convention, the IATA PADIS connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound IATA PADIS documents, or as the following step in the flow when receiving inbound IATA PADIS documents. The interaction between the XML Map connector and the IATA PADIS connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the IATA PADIS connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the IATA PADIS connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the IATA PADIS document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## IATA PADIS Acknowledgments The following sections detail the two types of IATA PADIS acknowledgments (ACKs), and how Arc expects, processes, and generates ACKs. ### Technical versus Functional Acknowledgments Technical ACKs, sometimes called interchange ACKs, are an indication that an interchange has taken place between the two parties, although not necessarily that any individual message has been exchanged. These serve as a receipt to the sender indicating that the PADIS message was successfully received, but it does not specify whether there were any issues with processing the message content. Functional ACKs are an indication that an interchange has been processed by the receiving party. They can report acceptance, acceptance with issues, or rejection of the received document. These serve as both a receipt that an interchange was successfully received and that it was fully processed. ### Expecting, Processing, and Generating Acknowledgments #### Expecting Acknowledgements IATA PADIS connectors operating in XML to IATA mode can be configured so that technical and/or functional ACKs are expected for a message. When one or both **Technical acknowledgment (CONTRL)** and **Functional acknowledgment (CONTRL)** are checked in the [Acknowledgments](#acknowledgments) section of the Settings tab, the connector maintains a Pending ACK status for a transmission until the appropriate ACKs have been returned and processed. This means that the connector status can be used to determine whether the recipient has confirmed that they received the interchange. See the [EDIFACT connector](./edifact) for a detailed description of this feature. The IATA PADIS connector works the same way. When EDI messages that require an acknowledgement have not received that acknowledgement, a Pending-Timeout header is added to the message. The message has a *Pending* status while it waits for its ACK. If it does not receive the ACK within the allotted time, the Pending-Timeout header is populated with the date and time when the message status changed to *Error*. When the ACK is received and the message is updated to *Success*, a new header is added with an empty string value. #### Processing Acknowledgments In a typical flow, PADIS ACKs arrive at an IATA PADIS connector operating in IATA to XML mode. This IATA PADIS connector can be configured to automatically route any received acknowledgments to the IATA PADIS connector that originally generated the document being acknowledged. Routing ACKs between IATA PADIS connectors is configured visually on the [Flows](../flows/flows) canvas by dragging the gray dot at the bottom of the IATA PADIS connector in IATA to XML mode onto the IATA PADIS connector that is in XML to IATA mode. Once the connector in XML to IATA mode receives the routed ACK, it pairs the ACK to the original message and changes its state from Pending ACK to Sent. #### Generating Acknowledgements When an IATA PADIS connector in IATA to XML mode receives a message and generates the corresponding XML, it can automatically generate CONTRL acknowledgments for the received message. To accomplish this, check **Technical acknowledgment (CONTRL)** and/or **Functional acknowledgment (CONTRL)** in the [Acknowledgments](#acknowledgments) section of the Settings tab. These acknowledgments must be routed to another IATA PADIS connector (in XML to IATA mode) to finalize the ACK. The connector to which the ACK is routed applies interchange headings and passes the ACK along to the next connector in the flow like any other IATA PADIS message. To route the ACK appropriately, drag the gray dot at the bottom of the IATA PADIS connector in IATA to XML mode onto the IATA PADIS connector that is in XML to IATA mode. See the [Generating Acknowledgement](./edifact#generating-acknowledgements) for a detailed description of this feature in the EDIFACT connector. The IATA PADIS connector works the same way. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# ICAP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/icap The ICAP connector enables integration with Internet Content Adaptation Protocol servers for real-time security scanning and content adaptation services. The ICAP connector enables seamless integration with Internet Content Adaptation Protocol (ICAP) servers in CData Arc. This connector facilitates content adaptation services, particularly real-time security scanning, by acting as an intermediary between your application and ICAP-compliant servers. ## Key Capabilities * Real-time content adaptation and security scanning * Support for both request modification (REQMOD) and response modification (RESPMOD) modes * TLS encryption support for secure communications * Customizable HTTP encapsulation for legacy system compatibility * Flexible authentication options including private certificate support * Functions as a Transform step in the flow. Files that are flagged as needing attention yield errors, while other files proceed down the flow as usual. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **ICAP URI** | (Required) The ICAP Uniform Resource Identifier (URI) of the ICAP server to connect to. Must be in this format: `icap://:/`. | | **Allow ArcScript in ICAP URI** | Allows for the evaluation of ArcScript expressions in the URI. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_arcscript_editor.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=05d3576b0141d612fcc6e83469592e28)), which enables you to build dynamic request URIs using ArcScript. | | **ICAP Headers** | Optional headers for specialized server requirements. Use this to add custom ICAP request headers that you want to include in the outgoing ICAP request. | | **Allow ArcScript in Headers** | Allows for the evaluation of ArcScript expressions in the headers before the query is issued. For example, `[_ \| now('yyyy-MM-dd')]` evaluates the current date, and `[_message.header:headername]` evaluates the headername header from the input message context. | | **TLS Configuration** | Check this to enable TLS encryption for the ICAP connection. | | **TLS Server Certificate** | The public key certificate used to verify the identity of an TLS/SSL server. This is only necessary if the ICAP server has enabled TLS. Set it to `Any Certificate` to unconditionally trust the ICAP server's identity. Only relevant when **TLS Configuration** is enabled. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### TLS Client Authentication | Setting | Description | | - | - | | **Private Certificate** | Select, create, or upload the TLS certificate to use for authentication. | | **Certificate Password** | The password required to access the **Private Certificate**. | #### Other Settings | Setting | Description | | - | - | | **ICAP Mode** | Specify the type of ICAP transaction: use REQMOD for request modification or RESPMOD for response modification. The default is REQMOD. | | **Preview size** | Controls the size of the message body preview sent to the ICAP server. `Automatic` makes your client query the server using an OPTIONS request to determine the optimal preview size. Alternatively, you can specify an exact number (in bytes) to use for the preview size. | | **Chunk size** | The maximum size (in bytes) of each data chunk sent after the preview phase (or for requests without a preview). The default is 16KB. | | **Timeout (seconds)** | The length of time in seconds that the connector waits for a connection response before throwing a timeout error. The default is 60 seconds. | | **Response Headers** | A comma-separated list of ICAP response header names (such as `X-Infection-Found,X-Scan-Result`) to be extracted and added as metadata to the processed message. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as ICAP transactions. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Incoming responses appear on this tab. ### Events Tab | Event | Description | | - | - | | **Response** | This event lets you extend the way the connector interprets replies from the ICAP server. By default, the connector finalizes messages based on the ICAP server's status code. However, since ICAP servers can also include scan details in response headers, the Response event lets you apply custom logic to those headers for more granular control. No output is created from this event; instead, you can use the `arc:throw` keyword to block the file (for example, if a particular header value indicates a failed scan). | | **Before Send** | This event fires after the connector has fully prepared a file to send/process a file, but before the sending/processing actually occurs. | | **After Receive** | This event fires after the connector has received a file. | The following are available in the Response event: * StatusCode: The ICAP response status code (such as 200, 204, or 4xx) * \_response: The ICAP response. You can use `[_response.header:xxx]` to get a specific header. * \_message: To refer to the input message (headers and body), and adding headers to the output. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# JSON Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/json The JSON connector converts XML data into JSON and JSON data into XML with automatic format detection and minimal configuration required. JSON connectors support converting XML data into JSON and JSON data into XML. ## Key Capabilities * Bidirectional JSON to XML conversion with automatic format detection * Minimal configuration required (the connector typically works with the default configurations) * Configurable attribute prefixes and text property handling for accurate translations * Custom formatting options including minification and root element naming * Type conversion support with JSON type annotations for precise data handling ## Overview The JSON connector enables conversion between JSON and XML without requiring much configuration. The connector automatically detects whether input files are JSON or XML, and then the connector translates the file to the other format. The connector supports configurable syntax for distinguishing attributes and ensuring that the output is valid XML or JSON. ## Connector Settings This section contains all of the configurable connector properties. ### Settings Tab #### Connector Settings *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Root Element Name** | When converting from JSON to XML, this value is used as the name of the root element in the resulting XML document. This ensures that the output XML has a single root element regardless of the structure of the input JSON. | | **Attribute Prefix** | The prefix is used to identify attributes in JSON. When converting from JSON to XML, the connector interprets values with this prefix as attributes in the resulting XML. When converting from XML to JSON, the connector adds this prefix to attribute values in the resulting JSON. See the [Examples](#examples-2) section for more information. | | **Text Property Name** | When a JSON object has attribute values and a non-attribute value, this name identifies the non-attribute value. When converting from JSON to XML, values associated with this name are set in the text of the resulting XML element. When converting from XML to JSON, the text in the element is set as the value for this name. See the [Examples](#examples-2) section for more information. | | **Minify** | When enabled, this option instructs the connector to reduce the whitespace in the resulting XML or JSON document. This option conserves space, but it can make the output more difficult for humans to read. | #### Other Settings | Setting | Description | | - | - | | **Invalid XML Name Prefix** | When a JSON element has a name that would not be valid XML, the connector adds this prefix to the beginning of the XML element name. For example, a JSON element named `10price` must be modified to translate it to XML. Although a JSON element name can start with a number, XML element names cannot. If you set this field to `num`, the `10price` JSON element becomes `num10price` in the XML output. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Conversion Examples Examples of converting between XML and JSON with the JSON connector are below. ### XML to JSON XML input: ```xml theme={null} Aesop Tortoise and the Hare Tortoise Hare Persistence over speed ``` JSON output: ```json theme={null} { "Fable": { "Author": { "@alive":"false", "#text":"Aesop" }, "Title":"Tortoise and the Hare", "Characters": { "Character": [ { "@id":"1", "@slow":"true", "#text":"Tortoise" }, { "@id":"2", "@slow":"hardly", "#text":"Hare" } ] }, "Moral":"Persistence over speed" } } ``` ### JSON to XML JSON input: ```json theme={null} { "Catch": [ { "type": "salmon", "quantity": "2", "weight": "20.2 lbs" }, { "type": "pike", "quantity": "21", "weight": "521 lbs" } ], "Lake": "Boulder", "Weather": "Clear", "808s": "Loud" } ``` XML output: ```xml theme={null} salmon 2 20.2 lbs pike 21 521 lbs Boulder Clear <_808s>Loud ``` ## JSON Type Conversions When converting from XML to JSON, the connector treats all XML elements as strings except in two cases: * If the connector detects multiple elements or values in the same XML structure in the XML input, it treats them as an array. * If you apply an XML attribute to specify the JSON output format, this attribute overrides the default string behavior. These attributes are listed in the table below. | Attribute | Description | | - | - | | **json:array="true"** | If you have a single-element array, you can apply this attribute to ensure that the connector treats it as an array instead of a string. | | **json:type="number"** | Instructs the connector to treat the XML element as a number. | | **json:type="boolean"** | Instructs the connector to treat the XML element as a boolean. | | **json:type="object"** | Instructs the connector to treat the XML element as an object. | | **json:type="null"** | The connector treats empty XML elements as strings of length zero. This command forces the output to be `null` instead of an empty string. | | **xsi:nil="true"** | Same functionality as **json:type="null"**. | # Label Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/label Configuration and usage guide for the CData Arc Label connector. The Label connector generates Zebra Programming Language (ZPL) labels based on a template, which is filled in at runtime using XML as input. ## Key Capabilities * ZPL label generation from XML input with template-based customization * Compatible with XML Map connector for automated label creation workflows * Dynamic field substitution using `${...}` syntax for thermal printer output ## Overview The Label connector converts input XML into ZPL files based on a predefined XML template, which enables automated and consistent label creation for printing on Zebra thermal printers. It is compatible with the [XML Map connector](./xml-map/xml-map) as well as raw XML input. However, the primary use case is for XML to be mapped to the ZPL structure represented in XML using a flow that goes from an XML Map to a Label connector. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to establishing the connection.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Template Tab The ZPL file to use as a template. When you create a Label connector, Arc pre-fills this field with a ZPL template to use as a starting point. See [Generating a ZPL Document](#generating-a-zpl-document) for details. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Check this to have the connector automatically send files when they are ready. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Generating a ZPL Document The Label connector reads an XML input file into the ZPL template specified on the [Template](#template-tab) tab, and uses the data from the template to generate ZPL labels. ### Default ZPL Template Arc pre-fills the Template field with a ZPL template to use as a starting point. This documentation does not go into detail on how to use the ZPL language: you need to learn that outside of Arc. You can modify other aspects and dynamic fields of the ZPL template as necessary, but here is the default content: ``` ^XA ^FX Top section with logo, name and address. ^CF0,60 ^FO50,50^GB100,100,100^FS ^FO75,75^FR^GB100,100,100^FS ^FO93,93^GB40,40,40^FS ^FO220,50^FDIntershipping, Inc.^FS ^CF0,30 ^FO220,115^FD1000 Shipping Lane^FS ^FO220,155^FDShelbyville TN 38102^FS ^FO220,195^FDUnited States (USA)^FS ^FO50,250^GB700,3,3^FS ^FX Second section with recipient address and permit information. ^CFA,30 ^FO50,280^FD${name}^FS ^FO50,320^FD${address}^FS ^FO50,360^FD${address2}^FS ^FO50,400^FD${city}^FS ^FO175,400^FD${state}^FS ^FO225,400^FD${zip}^FS ^FO50,440^FD${country}^FS ^CFA,15 ^FO600,300^GB150,150,3^FS ^FO638,340^FDPermit^FS ^FO638,390^FD123456^FS ^FO50,500^GB700,3,3^FS ^FX Third section with barcode. ^BY5,2,270 ^FO100,550^BC^FD${barcode}^FS ^FX Fourth section (the two boxes on the bottom). ^FO50,900^GB700,250,3^FS ^FO400,900^GB3,250,3^FS ^CF0,40 ^FO100,960^FDCtr. X34B-1^FS ^FO100,1010^FDREF1 F00B47^FS ^FO100,1060^FDREF2 BL4H8^FS ^CF0,190 ^FO470,955^FDCA^FS ^XZ ``` The elements using the `${...}` syntax (such as `${name}` in the content above) are what Arc pulls from the input XML passed into the connector. When you put an XML Map connector in front of a Label connector in a flow, the destination side of the mapping shows these fields, as shown in the following image. ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/label_destination_mapping.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=f82aebc678c8d03e2415ad865092d75d) If you map XML to these elements and feed it through the flow, the Label connector output (when viewed with a tool that reads .zpl files) contains the same ZPL data with the values filled in: ``` ^FO50,280^FDCharlie^FS ^FO50,320^FD123 Address Street^FS ^FO50,360^FDUnit 2^FS ^FO50,400^FDGotham^FS ^FO175,400^FDNY^FS ^FO225,400^FD12345^FS ^FO50,440^FDUnited States^FS ... ^FO100,550^BC^FD02271994^FS ``` Finally, when you run the ZPL through a reader to make the label, it looks like this: ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/label_zpl.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=7a91b1fcbb7e7226697d690ed7b3e629) ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# CSV Map Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/map Transforms flat XML files into CSV, TSV, and PSV formats using a visual Designer or ArcScript Code view for complex mappings. The CSV Map connector uses ArcScript to transform flat XML files into CSV, TSV, and PSV file formats. A flat XML file is one where, in addition to the root element, the depth is not more than two levels. ## Key Capabilities * Transforms flat XML files into CSV, TSV, and PSV formats using ArcScript templating * Visual Designer for simple mappings with automatic field matching and Code view for complex transformations * XPath navigation for extracting values from nested XML structures with dynamic template generation ## Overview The CSV Map connector has two modes of operation: the visual **Designer** and the scripting-focused **Code** view. The Designer enables you to visually establish simple relationships between document types. More complicated translations require the Code view, which leverages the ArcScript scripting language to create a mapping template between XML and a flat file structure. To begin, from the **Settings** tab, upload a sample **Source File** and **Destination File**. If the structure of both sample files allows for designer-driven mapping, the visual Designer is populated with the source and destination fields. The application also attempts to populate the Code window with as much of the destination structure as it can. ## Connector Configuration This section contains all of the configurable connector properties. ### Mapping Tab *Upload files to initiate design-driven mapping of source and destinations fields.* Click **Upload File**, then choose your source and destination files. * The **Source File** represents the XML structure of input documents. Any files processed by the connector should have a matching XML structure. To support designer-driven mapping, this file must be a simple XML, CSV, PSV, or TSV structure. * The **Destination File** file represents the structure of output documents. To use designer-driven mapping, this file must be a simple XML, CSV, PSV, or TSV structure. #### Mappings This section defines the mapping relationship between input and output files. The Designer view can map simple XML, CSV, PSV, and TSV structures, and the Code view can map more complicated XML structures. See [Using the Designer](#using-the-designer) and [Using Code View](#using-the-code-view) for details. ### Settings Tab #### Connector Configuration *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Advanced Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Using the Designer The visual designer is only available for CSV, PSV, TSV, and simple XML files. Below is an example of a simple XML document as the source file: ```xml theme={null} 124455 Bob Smith ``` This source file might be paired with the following CSV document as the destination file: ```csv theme={null} id,first name,last name,time 124455,Bob,Smith,2017-07-18T17:38:53-04:00 ``` When these files are configured as the source and destination files, the designer displays rows in the **Source** column for each row in the source file (actor\_id, first\_name, and last\_name). For each field in the source, use the dropdown to select the appropriate **Destination** column. The application automatically attempts to match the source and destination fields based on the column names. In the above example, the *actor\_id* row would automatically be matched with the *id* column. ![Mapping Designer](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/map_connector_designer.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=ea51fb006e664d482c3ee78eaff8559f) ## Using the Code View The Code view provides the ability to generate the mapping template manually using ArcScript. This allows for more granular control over the possible input and output formats. The code view defines how the destination file looks once rendered, with ArcScript elements dynamically filling in the template with values from the source document. ArcScript elements all start with an *arc* prefix: for example, ``. Any content in the code view that is not ArcScript is included as part of the output file. ### Navigating XML Input ArcScript supports navigating complicated XML structures to parse out values from the Source File. The [xmlDOMSearch](../scripting/operations/op-xml-dom-search) operation takes an xpath as input and loads the XML structure at the given path. This operation loops for each instance of the xpath found in the source document: to load the entire document and avoid looping, provide the root element of the XML source as input to [xmlDOMSearch](../scripting/operations/op-xml-dom-search). For example: ```xml theme={null} ``` Once the XML document is loaded, the [xpath formatter](../scripting/value-formatters/function-formatters#xpath-xpath) supports reading out values from the XML at the specified xpath. This formatter accepts absolute xpaths as well as xpaths relative to the path loaded by [xmlDOMSearch](../scripting/operations/op-xml-dom-search). The xpath formatter, like all formatters, can only be used in square brackets \[]. For example: ```xml theme={null} [xpath('name')] ``` Take the following simple XML as an example input to the above script: ```xml theme={null} Stephen King Kurt Vonnegut ``` In this example, the [xmlDOMSearch](../scripting/operations/op-xml-dom-search) operation loops over each `author` element in the `root`. Within each `author` loop, the value from the `name` element is read as content in the output file. The output file in this case would look like this: ``` Stephen King Kurt Vonnegut ``` ### Templating Output Data The Code view serves as a template of the output file, and ArcScript fills in the values to the defined template. As a simple example, if data is formatted as comma-separated values, these commas can be included as part of the Code view to provide structure to the ArcScript functions. Therefore, you can generate more complex CSV files from more complicated XML structures than the Designer mode allows. For example, take the following XML, which has nested elements that prevent the use of Designer mode: ```xml theme={null} 12445 Viggo Mortensen 12522 Gal Gadot ``` The nested elements can be retrieved by passing the appropriate paths to ArcScript's [xpath formatter](../scripting/value-formatters/function-formatters#xpath-xpath). These formatters can be placed in commas and after header names to provide the desired CSV structure, and an additional [csvescape formatter](../scripting/value-formatters/encoding-decoding-string-formatters#csvescape) ensures that the values are properly quoted: ```xml theme={null} id,first name,last name [xpath('actor_id') | csvescape],[xpath('name/first_name') | csvescape],[xpath('name/last_name') | csvescape] ``` You can generate a wide range of text files with this mix of templating data and ArcScript calls. ### Additional Scripting Features Since full ArcScript is available in Code view, you might want to leverage ArcScript [Operations](../scripting/operations/operations) in the template. For example, if the source file only includes the Id of an item in the database, but you need the actual item name, you can use the [dbQuery](../scripting/operations/op-db-query) operation to look up the name for the corresponding Id. ArcScript also supports conditional logic in a mapping template. The `arc:if` keyword is one of many keywords available to help with performing conditional logic in templates. For example, if the source file contains information about customers in QuickBooks, you might want to perform different business logic for customers with an outstanding balance versus customers who have paid in full. A simple example of this use case might look like the following: ```xml theme={null} id,first name,last name, paid in full [xpath('customer_id') | csvescape],[xpath('first_name') | csvescape],[xpath('last_name') | csvescape],[Customer.paidinfull] ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# MFT Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mft-connectors CData Arc MFT connectors extend core file-transfer capabilities with secure protocols, cloud storage providers, and industry-standard B2B messaging. MFT connectors extend the file-transfer features of the Core connectors by providing the following functionality: * Secure peer-to-peer file transfer protocols such as AS2 and AS4 * Server-side support for FTP and SFTP to host incoming connections * A browser-based partner mailbox for self-service file exchange * File uploads and downloads from cloud storage providers such as Amazon S3, Box, Dropbox, Google Drive, OneDrive, Wasabi, and others * Client-side and server-side support for OFTP, GISB, and RNIF for exchanging business documents * OpenPGP encryption and decryption for end-to-end security when transferring files Arc supports modern security standards that include: * TLS 1.0, TLS 1.1, TLS 1.2, and TLS 1.3 * AES and 3DES encryption * SHA1, SHA256, SHA384, and SHA512 signatures The Arc AS2 and AS4 implementations are certified by the [Drummond Group](https://www.drummondgroup.com/certified-products/b2b-interoperability/), and the Arc OFTP implementation is certified by the Odette Organization. # Microsoft 365 Email Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email Connect Arc to Microsoft 365 Email for data integration and synchronization The Microsoft 365 Email connector lets you send and receive email through the Microsoft Graph API using the CData Office365 driver. Unlike the [Email Send](../email-send) and [Email Receive](../email-receive) connectors, which support SMTP and IMAP for accounts such as Gmail, Yahoo, and on-premises Exchange, this connector is designed specifically for Microsoft 365 and supports app-only authentication for shared and departmental mailboxes. ## Key Capabilities * App-only authentication for shared and departmental mailboxes, using Azure Service Principal credentials without per-user OAuth consent * Outlook categories as non-destructive processing markers: stamp received messages with a server-side category and filter on it in subsequent polls instead of deleting or moving them * Rich server-side receive filters on folder, read status, importance, attachments, and received date, plus client-side filtering on Outlook categories * Send on behalf of or as another user or shared mailbox, with proper Microsoft Graph permissions * Full folder management: create processing folders, move messages after receive, or query folders by name or Id Follow the steps below to connect Arc to Microsoft 365 Email. ## Establish a Connection To allow Arc to use data from Microsoft 365 Email, you must first establish a connection. There are two ways to do this: * Add a Microsoft 365 Email connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Microsoft 365 Email**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Microsoft 365 Email can be reused across multiple Microsoft 365 Email connectors. Follow these steps to finish creating your connection: 1. Enter the following connection information in the **Add Connection** modal: * **Name**: The static name of the connection. Set a name of your choice. * **Type**: The connector name. This field is always set to Microsoft 365 Email. * **Auth Scheme**: The authentication scheme to use. Options are **AzureAD**, **AzureMSI**, **AzureServicePrincipal** (the default), and **AzureServicePrincipalCert**. See [Authentication Schemes](#authentication-schemes) for details on each scheme and its required fields. * **Default Groups**: The default group context when accessing group-scoped resources in Microsoft Office 365. * **Default User**: The default user context when accessing user-scoped resources in Microsoft Office 365. * **Message Content Type**: Whether to return message bodies in HTML format or as plain text. * **Group Id**: The Id of a Microsoft Office 365 group whose data you want to access. Optional. * **User Principal Name (UPN)**: The UPN or object Id of the mailbox this connector acts on, for example `support@xyzcorp.com`. Required for app-only schemes (AzureServicePrincipal, AzureServicePrincipalCert, and AzureMSI). Leave empty when using AzureAD to act on the signed-in user's own mailbox. 2. Enter any additional fields required for your chosen auth scheme. See [Authentication Schemes](#authentication-schemes) below. 3. Click **Connect** (AzureAD only) or **Test Connection** (all other schemes) to verify that Arc can connect with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. Select the newly created connection from the **Connection** drop-down list on the **Settings** tab. 6. Click **Save Changes**. ## Authentication Schemes The app-only schemes (AzureServicePrincipal, AzureServicePrincipalCert, and AzureMSI) require an application registration in the [Microsoft Entra admin center](https://entra.microsoft.com). The registration must be granted the **Mail.ReadWrite** and **Mail.Send** application permissions, both with tenant admin consent, before Arc can connect. Mail.ReadWrite covers Receive, the post-receive actions, and folder lookups; Mail.Send covers Send. Mail.Read is sufficient in place of Mail.ReadWrite only if you never use the post-receive actions. AzureAD uses delegated permissions and requires a redirect URI in the registration instead. ### AzureAD Delegated user authentication. The user signs in through a browser and grants access; tokens are stored and refreshed automatically. Use this scheme to act on the signed-in user's own mailbox. Leave **User Principal Name (UPN)** empty when using this scheme. No additional fields beyond the common fields above are required. After entering them, click **Connect**. A new browser tab opens for signing into your Microsoft account. Log in, grant the requested permissions, and return to Arc. ### AzureMSI Managed Service Identity authentication for Arc instances running on Azure infrastructure (virtual machines, App Service, or AKS pods with workload identity). The connector acquires a token from the host's managed identity automatically, without requiring credentials. Set **User Principal Name (UPN)** to the mailbox you want to access. No additional fields beyond the common fields above are required. ### AzureServicePrincipal Application-only authentication using a client secret. Does not require per-user OAuth consent, making it suitable for shared and departmental mailboxes. Set **User Principal Name (UPN)** to the mailbox you want to access. **OAuth** * **Azure Tenant** (required): The Id or primary domain of the Microsoft Entra tenant that contains the mailbox, for example `contoso.onmicrosoft.com`. * **OAuth Client Id** (required): The client Id assigned to your Azure AD app registration. * **OAuth Client Secret**: The client secret assigned to your Azure AD app registration. ### AzureServicePrincipalCert Application-only authentication using a certificate instead of a client secret. Use this scheme when your organization requires certificate-based credentials. Set **User Principal Name (UPN)** to the mailbox you want to access. **OAuth** * **Azure Tenant** (required): The Id or primary domain of the Microsoft Entra tenant that contains the mailbox, for example `contoso.onmicrosoft.com`. * **OAuth Client Id** (required): The client Id assigned to your Azure AD app registration. **JWT OAuth** * **OAuth JWT Cert**: The certificate to use for signing JWT assertions. * **OAuth JWT Cert Password**: The password for the OAuth JWT certificate. * **OAuth JWT Cert Subject**: Identifies the subject of the OAuth JWT certificate used to locate a matching certificate in the store. Supports partial matches and the wildcard `*` to select the first certificate. ## Select an Action After establishing a connection to Microsoft 365 Email, you must choose the action that the Microsoft 365 Email connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./ms365email-upsert) | Inserts or updates Microsoft 365 Email data. By default, if a record already exists in Microsoft 365 Email, an update is performed on the existing data in Microsoft 365 Email using the values provided from the input. | End | | [Lookup](./ms365email-lookup) | Retrieves values from Microsoft 365 Email and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Microsoft 365 Email. It should be formatted as a SQL query against the Microsoft 365 Email tables. | Middle | | [Lookup Stored Procedure](./ms365email-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./ms365email-select) | Retrieves data from Microsoft 365 Email and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./ms365email-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Receive](./ms365email-receive) | Receives email from a Microsoft 365 mailbox and passes it down the flow. | Beginning | | [Send](./ms365email-send) | Sends messages from the flow as email through Microsoft 365. | End | ## Advanced Tab ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are specific to the Microsoft 365 Email connector and the CData Office365 driver it relies on. Some settings only appear based on the selected action; see the documentation page for the action you are configuring for details. | Setting | Description | | - | - | | **Log Modules** | Specifies the core modules to include in the log file. Use a semicolon-separated list of module names. By default, all modules are logged. | | **Location** | Specifies the location of a directory containing schema files that define tables, views, and stored procedures. Depending on your service's requirements, this can be expressed as either an absolute path or a relative path. | | **Browsable Schemas** | Optional setting that restricts the schemas reported to a subset of all available schemas. For example, `BrowsableSchemas=SchemaA,SchemaB,SchemaC`. | | **Tables** | Optional setting that restricts the tables reported to a subset of all available tables. For example, `Tables=TableA,TableB,TableC`. | | **Views** | Optional setting that restricts the views reported to a subset of the available tables. For example, `Views=ViewA,ViewB,ViewC`. | | **Reset Schema** | Resets the cached schema, forcing the connector to retrieve fresh schema information from Microsoft Office 365 on the next operation. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Directory Retrieval Depth** | Specifies how far down a table's subdirectories should be scanned to retrieve results. If not explicitly set, the driver uses a depth of 5 sublevels below the root (default). | | **Max Rows** | Specifies the maximum number of rows returned for queries that do not include either aggregation or GROUP BY. | | **Other Driver Settings** | Specifies advanced connection properties for specialized scenarios. Use this property only under the guidance of our Support team to address specific issues. | | **Pagesize** | Specifies the maximum number of records per page the provider returns when requesting data from Microsoft Office 365. | | **Pseudocolumns** | Specifies the pseudocolumns to expose as table columns, expressed as a string in the format `TableName=ColumnName;TableName=ColumnName`. | | **Timeout** | Specifies the maximum time, in seconds, that the provider waits for a server response before throwing a timeout error. | | **Use Client Side Paging** | Toggles the CData ADO.NET Provider for Microsoft Office 365's use of client-side paging. When enabled, the connector retrieves all messages reliably but also returns calendar event messages alongside email; messages where `IsEventMessage` is true are automatically excluded from email results. | | **User Defined Views** | Specifies a filepath to a JSON configuration file that defines custom views. The provider automatically detects and uses the views specified in this file. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./ms365email-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Lookup Caching** | When enabled, queries are performed against a local cache before they are executed directly against the data source (when no records are returned from the cache). When disabled (the default), queries are performed directly against the data source. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Microsoft 365 Email Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-ac Advanced configuration options for the Microsoft 365 Email connector in Arc. The **Advanced** tab allows you to configure advanced properties of the Microsoft 365 Email connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Microsoft 365 Email connector models Microsoft 365 Email tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Microsoft 365 Email values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Microsoft 365 Email connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Microsoft 365 Email. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Microsoft 365 Email connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Microsoft 365 Email connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Microsoft 365 Email. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Microsoft 365 Email, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Microsoft 365 Email. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Microsoft 365 Email connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Microsoft 365 Email Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-esp Execute a stored procedure in Microsoft 365 Email from an Arc message in the flow. The Execute Stored Procedures action treats data coming into the Microsoft 365 Email connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Microsoft 365 Email connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Microsoft 365 Email Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-lookup Retrieve values from Microsoft 365 Email and insert them into an Arc message in the flow. The Lookup action retrieves values from Microsoft 365 Email and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Microsoft 365 Email connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Microsoft 365 Email connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Microsoft 365 Email connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Microsoft 365 Email connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Microsoft 365 Email tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Microsoft 365 Email connector can perform Lookup queries against a local cache instead of querying Microsoft 365 Email directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Microsoft 365 Email on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Microsoft 365 Email directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Microsoft 365 Email.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Microsoft 365 Email Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-lsp Execute a Microsoft 365 Email stored procedure to look up and insert values into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Microsoft 365 Email connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Microsoft 365 Email connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Microsoft 365 Email connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Microsoft 365 Email Receive Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-receive Receive email from a Microsoft 365 mailbox at the beginning of an Arc flow. The Receive action receives email from a Microsoft 365 mailbox and passes it down the flow. The Microsoft 365 Email Receive connector belongs at the beginning of an Arc [flow](../../flows/flows). On the connector **Action** tab, configure the following settings: | Setting | Description | | - | - | | **Folder** | The mail folder to receive messages from. Accepts a folder display name or a well-known folder name such as Inbox. Defaults to **Inbox**. | | **Download** | How the connector outputs each email. **Message** (the default) outputs the email as a single .eml file. **Attachments** outputs each attachment as a separate message; emails with no attachments produce no output while the watermark still advances. | ## Output Headers Each message passed down the flow carries the following headers: * `Microsoft365Email-Message-Id` * `Microsoft365Email-Subject` * `Microsoft365Email-Conversation-Id` * `Microsoft365Email-Categories` (absent when the message has no Outlook category) ## Receive Filters The following filters appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab) and restrict which messages the connector retrieves on each poll.
SettingDescription
**Read State**Restricts the messages retrieved to only unread or only read messages. Defaults to **Any**.
**Importance**Restricts the messages retrieved to a single importance level. Defaults to **Any**.
**Attachments**Restricts the messages retrieved to those with or without attachments. Defaults to **Any**.
**Category**Restricts the messages retrieved to those stamped with the specified Outlook category. Leave empty to receive messages regardless of category.
**Conversation Id**Restricts the messages retrieved to a single conversation thread. Each received message carries its conversation Id in the `Microsoft365Email-Conversation-Id` header, which is where this value comes from.
**Subject**Restricts the messages retrieved to those whose subject contains the specified value. Leave empty to receive messages regardless of subject.
**Sender**Restricts the messages retrieved to those sent from the specified email address. The address must match in full; a partial address or bare domain matches nothing. Leave empty to receive messages from any sender.
**Max Messages**The maximum number of messages to retrieve in a single poll. Messages are retrieved oldest first, so remaining messages are picked up on the following poll. Defaults to **50**.
**Days To Search**The number of days to look back when searching for messages. Defaults to **7**. When set to 0, no date limit is applied. When **Only Process New Messages** is enabled, this value is used only as a fallback when no prior receive watermark is available.
**Only Process New Messages**When enabled (the default), the connector tracks the last message received and will not re-download it on subsequent polls. The connector uses a stored watermark as the primary date filter and falls back to **Days To Search** when the watermark is unavailable. When disabled, every poll retrieves messages within the **Days To Search** window without tracking state.
## Post-Receive Action The following settings appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab) and control what happens to each message in the mailbox after it has been successfully received.
SettingDescription
**After Receiving**What to do with each message in the mailbox once it has been successfully received. Options are **None** (the default), **Mark As Read**, **Apply Category**, and **Move To Folder**. Applying a category or moving the message avoids deleting mail from the source mailbox.
**Category Name**The Outlook category applied to each received message, for example `Arc-Processed`. The category must already exist in the mailbox. Visible when **After Receiving** is set to **Apply Category**.
**Destination Folder**The mail folder that received messages are moved into. Accepts a folder display name or a well-known folder name. Visible when **After Receiving** is set to **Move To Folder**.
# Microsoft 365 Email Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-select Retrieve records from Microsoft 365 Email and use them to generate Arc messages in the flow. The Select action retrieves data from Microsoft 365 Email and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Microsoft 365 Email. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Microsoft 365 Email, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Microsoft 365 Email. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/ms365email/ms365email-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Microsoft 365 Email relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Microsoft 365 Email supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Receive Filters The following filters appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab) and restrict which messages the connector retrieves on each poll.
SettingDescription
**Read State**Restricts the messages retrieved to only unread or only read messages. Defaults to **Any**.
**Importance**Restricts the messages retrieved to a single importance level. Defaults to **Any**.
**Attachments**Restricts the messages retrieved to those with or without attachments. Defaults to **Any**.
**Category**Restricts the messages retrieved to those stamped with the specified Outlook category. Leave empty to receive messages regardless of category.
**Conversation Id**Restricts the messages retrieved to a single conversation thread. Each received message carries its conversation Id in the `Microsoft365Email-Conversation-Id` header, which is where this value comes from.
**Subject**Restricts the messages retrieved to those whose subject contains the specified value. Leave empty to receive messages regardless of subject.
**Sender**Restricts the messages retrieved to those sent from the specified email address. The address must match in full; a partial address or bare domain matches nothing. Leave empty to receive messages from any sender.
**Max Messages**The maximum number of messages to retrieve in a single poll. Messages are retrieved oldest first, so remaining messages are picked up on the following poll. Defaults to **50**.
**Days To Search**The number of days to look back when searching for messages. Defaults to **7**. When set to 0, no date limit is applied. When **Only Process New Messages** is enabled, this value is used only as a fallback when no prior receive watermark is available.
**Only Process New Messages**When enabled (the default), the connector tracks the last message received and will not re-download it on subsequent polls. The connector uses a stored watermark as the primary date filter and falls back to **Days To Search** when the watermark is unavailable. When disabled, every poll retrieves messages within the **Days To Search** window without tracking state.
## Post-Receive Action The following settings appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab) and control what happens to each message in the mailbox after it has been successfully retrieved.
SettingDescription
**After Receiving**What to do with each message in the mailbox once it has been successfully retrieved. Options are **None** (the default), **Mark As Read**, **Apply Category**, and **Move To Folder**. Applying a category or moving the message avoids deleting mail from the source mailbox.
**Category Name**The Outlook category applied to each retrieved message, for example `Arc-Processed`. The category must already exist in the mailbox. Visible when **After Receiving** is set to **Apply Category**.
**Destination Folder**The mail folder that retrieved messages are moved into. Accepts a folder display name or a well-known folder name. Visible when **After Receiving** is set to **Move To Folder**.
## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Microsoft 365 Email connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Microsoft 365 Email Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/ms365email_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **Microsoft 365 Email\_Select** connector pulls data from a Microsoft 365 Email connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Microsoft 365 Email\_Select** then passes down the flow from **CSV1**.
# Microsoft 365 Email Send Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-send Send email through Microsoft 365 at the end of an Arc flow. The Send action sends messages from the flow as email through Microsoft 365. The Microsoft 365 Email Send connector belongs at the end of an Arc [flow](../../flows/flows). On the connector **Action** tab, configure the following settings: | Setting | Description | | - | - | | **To** | A semicolon-separated list of recipient email addresses. Macros may be used to dynamically include information such as identifiers and timestamps. | | **CC** | A semicolon-separated list of email addresses to copy on the message. | | **BCC** | A semicolon-separated list of email addresses to blind-copy on the message. | | **Subject** | The subject line of the outgoing email. Macros may be used to dynamically include information such as identifiers and timestamps. | | **Send Message As** | Whether the file arriving in the flow becomes the body of the email (the default) or is attached to it. Microsoft Graph limits a single send request to 4 MB, so larger files cannot be attached. | | **Body Type** | Whether the message body is sent as HTML (the default) or as plain text. | The following settings appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab):
SettingDescription
**Send As**Sends the email as another user or shared mailbox. Under AzureAD, this requires the delegated **Mail.Send.Shared** permission. Under the app-only schemes, it requires the **Mail.Send** application permission plus Send As rights granted on the target mailbox by an Exchange administrator.
**Send On Behalf Of**Sends the email on behalf of another user. Recipients see both the sender and the mailbox the message was sent from.
# Microsoft 365 Email Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ms365email/ms365email-upsert Insert or update records in Microsoft 365 Email from an Arc message in the flow. The Upsert action updates or inserts Microsoft 365 Email data. By default, if a record already exists in Microsoft 365 Email, an update is performed on the existing data in Microsoft 365 Email using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Microsoft 365 Email Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Microsoft 365 Email. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/ms365email/ms365email-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## Advanced Tab Settings The following settings appear on the [**Advanced** tab](/26.4/self-hosted/en/connectors/ms365email/ms365email#advanced-tab):
SettingDescription
**Send As**Sends the email as another user or shared mailbox. Under AzureAD, this requires the delegated **Mail.Send.Shared** permission. Under the app-only schemes, it requires the **Mail.Send** application permission plus Send As rights granted on the target mailbox by an Exchange administrator.
**Send On Behalf Of**Sends the email on behalf of another user. Recipients see both the sender and the mailbox the message was sent from.
## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Microsoft 365 Email connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Microsoft 365 Email Upsert connector at the end: ![](/26.4/images/ms365email_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Microsoft 365 Email\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Microsoft 365 Email.
# Notify Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/notify The Notify connector sends simple notification emails to recipients defined in the application Alerts settings, with support for macros and ArcScript in message content. The Notify connector sends simple notification emails to the recipients defined on the application [Alerts](../getting-started/administration/settings/alerts) page. ## Key Capabilities * Simple email notification system using application-wide alert settings * Configurable subject lines and message bodies with macro and ArcScript support * Automatic email sending to predefined recipients from the Alerts configuration * Support for file attachments and both synchronous and asynchronous delivery ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Subject** | The subject line of the outgoing email message. | | **Message** | The body of the outgoing email message. ArcScript is supported, as are the following macros: %MessageId%, %ConnectorID%, %Filename%, %FilenameNoExt%, %EXT%, %ShortDate%, %LongDate%, %RegexFilename:\[regex]%, %DateFormat:\[format]%, %Header:\[headerName]% | > [!TIP] > > If you have not already specified notification settings on the [Alerts](../getting-started/administration/settings/alerts) page, click **Configure Notification Settings** to set them up. ### Advanced Tab | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Notify Operations In addition to the Operations provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except for two details: 1. They must be called through the `connector.rsc` endpoint. 2. They must include an auth token. For example, calling a connector operation using both of these rules might look something like this: ```xml theme={null} ``` The operation specific to the functionality of the Notify connector is listed below. ### notifySendEmail Sends email notifications. #### Required Parameters * **Message**: The message to send. #### Optional Parameters * **To**: A comma separated list of addresses for destinations. * **Subject**: The message subject. * **Attachment#**: File(s) to attach to the message. * **Async**: Send asynchronously. The allowed values are `true` or `false`. The default value is `true`. # OFTP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oftp Configuration and usage guide for the CData Arc OFTP connector, which sends and receives files via the Odette FTP protocol. The OFTP connector sends and receives files via the Odette FTP protocol. ## Key Capabilities * Odette FTP support for automotive and manufacturing industry secure file exchange * Bidirectional file transfer with certificate-based encryption, signing, and compression * Message routing through intermediary OFTP entities with SSID/SFID configuration * Automatic retry and resend logic with synchronous and asynchronous receipt handling ## Overview An OFTP connection is configured in two places. Configure the OFTP [Profiles](../getting-started/administration/profiles) page with an Odette identifier, settings for the local OFTP server, and other information that is global across all OFTP connections. Then configure individual OFTP connectors with connection settings specific to a single trading partner (a single OFTP entity). For information on routing an OFTP message through a separate OFTP entity (for example, a clearinghouse), see [Message Routing](#message-routing). ## Profile Configuration The OFTP profile must be configured before connections can be established with individual OFTP connectors. Click **Profiles > OFTP** on the navbar. ### OFTP Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **Odette Identifier (SSID Code)** | Your Odette identifier. This value is included in outgoing transmissions to identify the sender, and incoming messages must be addressed to this identifier. Odette identifiers are provided by the Odette organization. | | **Password (SSID Password)** | The password associated with the **Odette Identifier**. You can choose this value as long as your trading partners have the same value configured on their systems. | #### Server Settings *Settings related to the OFTP server implementation.* | Setting | Description | | - | - | | **Port** | The port on which the server listens for incoming connections. | | **SSL/TLS** | Whether SSL/TLS must be negotiated to connect to the server. | | **TLS Private Certificate** | If TLS/SSL is enabled, specify the TLS/SSL certificate required to verify the server's identity. | | **Certificate Password** | The password required to access the TLS/SSL certificate. | #### Personal Certificate *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Data Decryption Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in an OFTP transaction; a corresponding public key certificate is also generated with the same filename and a `.cer` extension. | | **Certificate Password** | The password required to access the **Data Decryption Certificate**. | #### Optional Certificate Settings *Optional settings to define parameters for a certificate.* | Setting | Description | | - | - | | **Auth Challenge Private Certificate** | A certificate with a private key used when challenged by the server to verify authenticity. Accepts PKCS#12 certificates in `.pfx` or `.p12` formats. | | **Certificate Password** | The private certificate password. | | **Sign Private Certificate** | A certificate with a private key used when signing sent messages. Accepts PKCS#12 certificates in `.pfx` or `.p12` formats. | | **Certificate Password** | The private certificate password. | | **Strong Certificate Verification** | Only allow strong certificate verification from trusted certificate authorities. | #### Advanced Settings | Setting | Description | | - | - | | **Inactivity Timeout** | The length of time (in seconds) to wait before closing inactive connections to the server. | #### Lockouts *Optional settings related to locking server access.* | Setting | Description | | - | - | | **Failed Attempts** | The number of unsuccessful login attempts allowed before the user is locked out. | | **Lockout Period** | The length of time (in minutes) that the user is locked out. | | **Time Check Period** | The length of time (in minutes) that records are kept of failed login attempts. | #### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature:
  • `localhost` cannot be modified or removed from the list.
  • Any IP addresses outside of the defined ranges are rejected.
  • Ranges are supported. For example, the entry `100.10.100.1-15` indicates that IP addresses between `100.10.100.1` and `100.10.100.15` are allowed.
  • Classless inter-domain routing (CIDR) notation is supported. For example, the entry `100.10.100.0/24` indicates that IP addresses between `100.10.100.0` and `100.10.100.255` are allowed.
  • Wildcard patterns are supported. For example, the entry `100.10.100.*` indicates that IP addresses beginning with `100.10.100` are allowed.
> [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Rotate Interval** The number of days to wait before creating a new log file.
**Log Delete Interval** The number of days to wait before deleting old log files.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to the remote OFTP entity to connect to.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Server Status** | The current state of the connector's underlying service: **Active** (the service is running), **Inactive** (the service is configured but is not currently running), **Not Configured** (the underlying profile or service has not been set up). Depending on whether you have already created the OFTP profile, click the **Configure OFTP Profile** or **View OFTP Profile** links to open the profile in a new tab. | | **Odette Identifier** | The Odette identifier for the remote OFTP entity (the trading partner's identifier). | | **Password** | The password associated with the partner's identifier. | | **Version** | The version of the OFTP protocol to use when exchanging files. | | **Remote Host** | The hostname or IP address of the remote OFTP server. | | **Port** | The port on which to connect to the remote OFTP server. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Use SSL/TLS** | Whether to negotiate SSL/TLS when connecting to the remote OFTP server. | | **Secure Odette Authentication** | Whether to perform application-layer encryption and decryption. To perform this authentication, you must specify both a signing and encryption certificate. | #### Connection Info *Settings related to the OFTP connection parameters.* | Setting | Description | | - | - | | **Virtual File Format** | The OFTP file format to use when transmitting files. | | **Virtual Outgoing File Security** | Whether to sign and/or encrypt outgoing messages. | | **Virtual Incoming File Security** | Whether to require signed and/or encrypted incoming messages. | | **Compression** | Whether to compress outgoing messages. | | **Signed Receipts** | Whether outgoing messages should request or require a signed receipt to be returned. | #### Trading Partner Certificates *Settings related to the public key certificates provided by the trading partner.* | Setting | Description | | - | - | | **Encryption Certificate** | The public key certificate used for encryption when sending messages. This certificate must be paired with the trading partner's private decryption certificate. The trading partner must provide this public key certificate when sharing their OFTP configuration details. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the partner's OFTP system requires HTTPS instead of HTTP. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server's identity. | #### Routing *Settings related to routing an OFTP message through a separate OFTP entity.* | Setting | Description | | - | - | | **Routing Partner** | If outgoing OFTP messages should be routed through an OFTP entity that is separate from the target OFTP entity, set this field to the OFTP connector that is configured to connect to the intermediary OFTP server. | For example, if the connector needs to send files to `serverA` and route the file through `serverB`, configure the connector to connect to `serverA` and set this field to an OFTP connector configured to connect to `serverB`. See [Message Routing](#message-routing) for more information. ### Advanced Tab #### Optional Certificates (PEM/CER Format) *Settings related to additional certificates to use for enhanced security.* | Setting | Description | | - | - | | **Auth Challenge Certificate** | The public key certificate to present when challenged by the server to verify authenticity. | | **Verification Certificate** | The public key certificate to use when verifying digital signatures on incoming messages. | | **Receipt Verification Certificate** | The public key certificate to use when verifying digital signatures on incoming receipts. | | **Rollover Certificate** | An additional certificate to use for authentication. This is useful when you need an overlap period while changing certificates. | | **Rollover Verification Certificate** | An additional certificate to use for signature verification. This is useful when you need an overlap period while changing certificates. | #### Alternate Local Profile *Settings that override the OFTP configuration on the Profile page. Setting an alternate local profile allows the use of different local certificates and identifiers for certain trading partners.* | Setting | Description | | - | - | | **SSID** | Your Odette identifier. This overrides the **Odette Identifier** on the Profiles tab. When set, the OFTP server uses the alternate SSID and password for this partner. When sending, this ID is used as the file sender and for session authentication, unless it is overridden by the relaying partner. If **SSID** and **SFID** are both set, the **SFID** is used when sending the file. | | **Password** | The local identifier password. | | **SFID** | When accepting incoming OFTP files, CData Arc processes messages in this connector as if they were relayed from the local OFTP profile. Unless this value is set, incoming network connections are greeted with the SSID and password from the OFTP profile. If **SSID** and **SFID** are both set, the **SFID** is used when sending the file. | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. This overrides the **Data Decryption Certificate** setting on the Profiles page. | | **Certificate Password** | The password required to access the local private certificate. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Use Profile Settings** | Use the private certificate on the Profiles page. | | **Private Certificate** | The private certificate presented during TLS client authentication. | | **Certificate Password** | The password required to access the TLS client certificate. | #### Automatic Certificate Identification The fields in this section let you specify patterns that allow the connector to automatically identify certificates. The default wildcard character in each field (\*) instructs the connector to not search for each pattern. #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Credit Count** | The maximum credit value to be sent in the initial connection (SSID command). | | **Encryption Algorithm** | The algorithm to use when encrypting outgoing messages. | | **Exchange Buffer Size** | The data exchange buffer size, in bytes, to be sent in the initial connection (SSID command). | | **Enforce Protocol Version** | Whether to accept protocol versions not configured for this partner. | | **File Description Map** | The map to set file description text for sending files. Use a semi-colon-delimited list of glob patterns for matching filenames. For example, `INVOICE*=This is an invoice;*ORDER*=This is an order`. | | **Max Record Size** | When the virtual file format is Fixed or Variable, this field determines the maximum size of each record. The default record size is 128. | | **File Hash Algorithm** | The file hash algorithm to use. | | **Receipt Hash Algorithm** | The receipt hash algorithm to use. | | **Received Filename Format** | Incoming files are written to the **Transactions** tab using this filename convention. You can use the following macros to ensure that unique filenames are generated: %VirtualFilename%, %VirtualFileDate%, %GUID%. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2. | | **Virtual Filename Map** | The map from local filenames to outgoing virtual filenames to use when sending files. Use a semi-colon-delimited list of glob patterns for matching filenames. For example, `INVOICE*=filename1;*ORDER*=filename2`. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send Automation** | Whether files arriving at the connector are automatically sent as OFTP messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connector retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous MDN receipt is not provided within the expected time frame. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous receipts are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | | **Receive** | Whether the connector should automatically make pull requests to receive messages from the trading partner. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The following settings are required to establish an outgoing OFTP connection: * Identifier (the trading partner's identifier) * Password (the password associated with the trading partner's identifier) * Remote Host * Port After establishing a connection, the appropriate trading partner certificates must be configured before files can be securely transferred. ## Send and Receive Files Once the OFTP profile and partner-specific OFTP connectors have been configured, files can be securely sent and received. ### Send Files In an OFTP connector, the **Transactions** tab displays the files to be sent to the target trading partner. If **Send Automation** is enabled on the [Automation](#automation-tab) tab, files that reach the Transactions tab of the connector are automatically packaged and sent. Access the log files for all transmissions by expanding the row associated with the transmitted file. The **Create Test Files** button lets you generate a simple series of test files to send to the trading partner. ### Resend and Retry An OFTP Resend is triggered when the trading partner is expected to return an asynchronous receipt, but fails to do so within the **Resend Interval** duration (60 minutes by default). The application then attempts to resend the transmission. The application continues resending the message until a receipt is received or the **Max Attempts (async)** is exhausted. A Retry is triggered when the protocol response from the trading partner's system indicates that the server has not received the transmission. This can indicate a networking or connectivity issue, which is often transient. The application retries the transmission every **Retry Interval** minutes until the transmission is received or the **Max Attempts** is exhausted. ### Receive Files In an OFTP connector, the **Transactions** tab displays the files that have been received by the application and routed to the connector. Files are routed to a specific OFTP connector based on the Odette identifier present in the incoming OFTP message. Expand each file row to display a list of available logs for the transmission. These files are available on the connector **Transactions** tab. If the connector is connected to other connectors in the flow, files are automatically moved from the **Transactions** tab of the OFTP connector to the **Transactions** tab of the next connector in the flow. ## Message Routing Each OFTP connector is configured to connect to a single OFTP entity. Sometimes using OFTP requires a routing scenario where a file originating at one OFTP entity (agent `A` in the image below) needs to pass through another OFTP entity (agent `B`) to reach the destination OFTP entity (agent `C`). ![OFTP Routing](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/oftp-relaying.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=8edfbc55e64e1a31740830272f032cfe) In this scenario, the Odette identifier of agent `B` is the SSID value during the transfer, and the Odette identifier of agent `C` is the SFID. In other words, the SSID identifies the 'middleman' server that the originator connects to, and the SFID identifies the target entity that this 'middleman' should route the OFTP message to. ### OFTP Routing in Arc To accomplish this routing scenario in Arc, configure one OFTP connector to connect to agent `B` (the 'middleman' server) and another OFTP connector to connect to agent `C` (the destination server). Set the **Routing Partner** field (in the [Routing](#routing) portion of the **Settings** tab) for the agent `C` connector to the OFTP connector that is configured to connect to agent `B`. Files to send out to the destination server should be processed by the OFTP connector configured to connect to agent `C`. The other OFTP connector (targeting agent `B`) is used behind-the-scenes to route the message, but does not directly process files in the Flow. ### SSID vs SFID If a partner provides a single identifier that they call an SFID, configure an OFTP connector to connect with this entity, and use the SFID as the SSID. If the partner provides an SFID value and an SSID value, the SFID corresponds to server `C` in the above scenario, and the SSID corresponds to server `B`. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# OneDrive Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/onedrive Configuration and usage guide for the CData Arc OneDrive connector, which uploads to and downloads from the OneDrive cloud storage service. The OneDrive connector uploads to and downloads from the OneDrive cloud storage service. ## Key Capabilities * Microsoft OneDrive cloud storage integration with OAuth authentication and automatic token refresh * Bidirectional file transfer functionality with folder-based organization and glob pattern filtering * Optional file deletion after download with overwrite protection * Optional caching to ensure only new or updated files are downloaded ## Overview Each OneDrive connector connects to a specified OneDrive storage account. Within the remote storage, you set a single folder as the upload target and a single folder as the download target. Files that reach the OneDrive connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specific file names and/or file extensions. Account authentication uses OAuth, so you must enter account credentials in an external site portal the first time the connector connects. The external portal generates an OAuth token that the connector uses to connect, then the connector handles token refresh automatically. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
Click the **Connect** button to authorize the connector with the storage service by redirecting to a site login portal. Once you enter the account credentials in the portal, the resulting authorization token is returned to Arc and stored for repeated use. #### Send *Settings related to uploading to the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | Files processed by the connector are uploaded to the folder specified here. | | **Overwrite** | Whether files that already exist in the remote folder should be overwritten or skipped. | #### Receive *Settings related to downloading from the remote storage.* | Setting | Description | | - | - | | **Remote Folder** | The remote folder from which files are downloaded. | | **File Mask** | A glob pattern that determines which files in the **Remote Folder** should be downloaded (for example, \*.txt). If you need to specify multiple file masks, use the **Receive Filter** in the Advanced Settings instead. | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **Timeout** | The duration the connector waits for a connection response before throwing a timeout error. | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection All that you need to establish a connection with the OneDrive connector is a set of OneDrive account credentials. Click the **Connect** button on the [Settings tab](#settings-tab) to be directed to a login portal site. Enter your account credentials in this portal to return an authorization token to the connector. This process is only necessary for the first time the connector connects to the storage service. ## Uploading and Downloading ### Upload To upload files, set the **Remote Folder** field in the [Send](#send) portion of the **Settings** page to the folder where files should be uploaded. Each OneDrive connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the remote folder should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Download To download files, set the **Remote Folder** field in the [Receive](#receive) portion of the **Settings** page to the folder where files should be downloaded. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the [Advanced tab](#advanced-tab). Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. After configuration is complete, files are downloaded according to the **Receive** [Automation](#automation-settings) settings, or you can manually click the **Receive** button on the **Transactions** tab. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# OpenPGP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/openpgp Configuration and usage guide for the CData Arc OpenPGP connector, which supports encryption, decryption, signing, and verification according to the Open Pretty Good Privacy standard. The OpenPGP connector supports encryption, decryption, signing, and verification according to the Open Pretty Good Privacy standard. ## Key Capabilities * Complete OpenPGP encryption, decryption, signing, and signature verification with multiple algorithm support * Encoding mode (encrypt/sign) and decoding mode (decrypt/verify) with compression options * Keyring management and ASCII armor encoding for readable encrypted data transmission ## Overview OpenPGP connectors are the primary way that Arc supports protecting data in a flow. OpenPGP connectors operating in **Encode** mode can encrypt and/or sign files, and OpenPGP connectors operating in **Decode** mode can decrypt files and/or verify signatures. Encryption and signature verification require a public OpenPGP key, and decryption and signing require a private OpenPGP key. These keys must be [created](#creating-keys) or imported into OpenPGP keyring files (`.gpg`) before you can use them with the application. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Operation** | Whether the connector is encoding or decoding incoming files. Encoding includes encrypting and signing data, and decoding includes decrypting data and verifying signatures. The connector settings change based on this setting. | #### Message Settings *Settings related to creating an OpenPGP message. Only available when encoding.* | Setting | Description | | - | - | | **Message Security** | Whether the connector should create an encrypted message, a signed message, or both a signed and encrypted message. | | **Compression** | Whether the connector should compress the message before encrypting and/or signing it. | | **Encryption Algorithm** | The symmetric algorithm to use when encrypting. | | **Signature Algorithm** | The hash algorithm to use when signing. | | **Compression Method** | The compression algorithm to use when compressing. | #### Keys *Settings related to the OpenPGP keys used by the connector. Encryption and signing are only available when encoding, while verification and decryption are only available when decoding.* | Setting | Description | | - | - | | **Encryption Key** | The user Id identifying the public key in a public keyring to use when encrypting. Import a public keyring file to view the available user Ids. | | **Signing Key** | The user Id identifying the private key in the secret keyring to use when signing. Import a secret keyring file to view the available user Ids. | | **Verification Key** | The user Id identifying the public key in the public keyring to use when verifying signatures. Import a public keyring file to view the available user Ids. | | **Decryption Key** | The user Id identifying the private key in the secret keyring to use when verifying signatures. Import a secret keyring file to view the available user Ids. | | **Passphrase** | When encoding: the passphrase for the selected private signing key. When decoding: the passphrase for the selected private decryption key. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **ASCII Armor** | Whether ASCII-encoding should be applied to OpenPGP messages generated by the connector. | | **Clear Signature** | Whether the OpenPGP signature should appear in clear text. Not applicable when encrypting messages. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send Automation** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Encoding When encoding files, configure each of the settings under **Message Settings**. These determine how the file is encoded. If encryption is required, specify a public encryption key in the **Encryption Key** field. If signing is required, specify a private signing key in the **Signing Key** field. To select a key in a keyring, import the keyring file then use the dropdown to choose a user Id. To sign with a private key, provide the **Passphrase** required to access the private key. You can enable the **ASCII Armor** option on the [Advanced tab](#advanced-tab) to ASCII-encode encrypted data so that it remains readable. You can use the **Clear Signature** option if the signature should appear in clear text (not possible when encrypting files). Once you set these options, files sent to the input directory of the OpenPGP connector are automatically encoded. ## Decoding When decoding files, the connector automatically attempts to determine what encryption and/or signature algorithms were applied, so you do not need to configure the connector for specific algorithms. If decryption is required, specify a private decryption key in the **Decryption Key** field (supply the private key that corresponds to the public key that was used to encrypt). If signature verification is required, specify a public verification key in the **Verification Key** field (supply the public key that corresponds to the private key used to sign). To select a key in a keyring, import the keyring file then use the dropdown to choose a user Id. To decrypt with a private key, provide the **Passphrase** required to access the private key. Once you set these options, files sent to the input directory of the OpenPGP connector are automatically decoded: encrypted files are decrypted, and signed files are verified. ## Creating Keys To create a key: * Select **Import/Export > Create Key** to begin creating a new OpenPGP key pair: * If the connector is in **Encode** mode, this is next to **Signing Key**. * If the connector is in **Decode** mode, this is next to **Decryption Key**. * Enter the following information: * **User Id**: Provide at least **First Name** or **Email** to create a key. The User Id for the key is comprised of the first name, last name, and email fields in the key creation wizard. * **Passphrase**: Enter a passphrase to protect the private key. The passphrase is used in the decrypt, encrypt, and sign operations. * **Key Encryption Algorithm** and **Key Signature Algorithm**: Select the encryption algorithm that corresponds to the desired strength of your encryption. Select the signature algorithm that corresponds to the desired length of the hash of the message. * Click **Create Key**. Keys are created in the `data/~Profiles/OpenPGP` folder relative to the Application Directory. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Common Errors ### Error: Unknown PGP Packet tag When attempting to decode a GPG message using the OpenPGP connector, you might get the error **Unknown PGP Packet tag** on the **Transactions** tab and in the logs. **Cause** The GPG message has been encrypted with the AEAD cipher. AEAD is a cipher that is [still in draft](https://datatracker.ietf.org/doc/html/draft-ietf-openpgp-crypto-refresh-04#section-5.16), and Arc does not yet support it. **Resolution** GPG messages encrypted in GPG 2.3.0 and later using keys created in GPG 2.3.0 and later need to be encrypted using the following options to disable the cipher: `--force-mdc --rfc2440 --encrypt` GPG packets encrypted in earlier releases or encrypted in GPG 2.3.0 or later using keys created in prior releases are not affected. # Partner Mailbox Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/partner-mailbox Configuration and usage guide for the CData Arc Partner Mailbox connector. The Partner Mailbox connector provides your trading partners with a private, browser-based mailbox for self-service file exchange, where they log in to download files staged for them and upload files to send. ## Key Capabilities * One mailbox per trading partner, each with its own credentials, URL, and folder tree * Self-service file exchange for partners who do not have typical MFT capabilities * Organized Send and Receive folder trees with drag-and-drop management * File retention controls for downloaded files, with re-download support * File extension filtering to control what partners can upload * Customizable page title and logo branding for the partner-facing mailbox page * Routing by folder using the standard `SubFolder` header, consistent with other MFT connectors ## Overview Each Partner Mailbox connector represents one trading partner's mailbox. The partner opens a dedicated web page, authenticates with a username and password, and can then download files that have been staged for them and upload files to send. Multiple trading partners require multiple connector instances. The [partner's mailbox page](#partner-mailbox-view) is organized into two sections: files staged for the partner to download (Send folders), and folders where the partner can upload files (Receive folders). Each section uses an independent folder tree that the admin manages in the connector configuration. Partner uploads enter the Arc flow as messages the same way files from any other connector do. Routing within the flow is handled by the `SubFolder` header, described in [How Routing Works](#how-routing-works). > [!NOTE] > > Currently, the web page is the only access method. SFTP and FTP access to the same mailbox are planned for a future release. ## Connector Configuration ### Mailbox Config Tab The **Partner Mailbox URL** displayed at the top of this tab is the address the partner uses to access their mailbox. The URL takes the form `.../connector///mailbox.rst`. Share this URL with the partner after your configuration is complete. #### Mailbox Authentication *Credentials the partner uses to log in to the mailbox page.* | Setting | Description | | - | - | | **User** | The username for the partner's mailbox account. Each mailbox has one account. | | **Password** | The password for the partner's mailbox account. | #### File Management – Partner Download *Settings that control how files staged for the partner are managed after download.* | Setting | Description | | - | - | | **When a partner downloads a file** | Whether to keep the file available for re-download or remove it after the first download. **Keep the file until the retention period passes** allows re-downloads until the file expires, while **Delete after download** removes the file after the first download. Setting **File Retention For Downloaded Files** to 0 has the same effect as **Delete after download**. | | **File Retention For Downloaded Files** | The number of days a downloaded file remains available for re-download. The default is 7 days. Retention is counted from the date of the first download, not from when the file was staged, and re-downloading does not reset the clock. Only applicable if **When a partner downloads a file** is set to **Keep the file until the retention period passes**. | #### File Management – Partner Upload *Settings that control which files partners are allowed to upload.* | Setting | Description | | - | - | | **File Extensions** | A glob pattern filter applied to files before they are accepted. For example, `*.txt` accepts only .txt files, and `-*.pdf` blocks .pdf files. Separate multiple file types by commas. Leave blank to accept all file types. | #### Partner Mailbox Page *Settings for the appearance of the partner-facing mailbox page.* | Setting | Description | | - | - | | **Page Title** | The title displayed at the top of the partner's mailbox page. | | **Partner Mailbox Logo** | An image to display as the logo on the mailbox page, along with a background color for the logo area. | #### Partner Mailbox Folders The folder configuration is split into two tabs: **Send Folders** for files staged for the partner to download, and **Receive Folders** for folders where the partner uploads files. Each tab has an independent folder tree that starts with one root folder. The **Section Header** field sets the heading displayed above that section on the partner's page. You can add, rename, move, and delete subfolders to any depth below the root. Folder names accept letters, numbers, and underscores only, and must be unique within the same parent (compared case-insensitively). You cannot delete the root folder. Hover over a folder, then use the tablet and pencil icon to rename it if necessary. Use the folder with a plus sign icon to add subfolders to the tree. You cannot delete the top-level folder, but you can delete subfolders by clicking the bin icon. See [Managing the Folder Tree](#managing-the-folder-tree) for details. ##### Send Folders | Setting | Description | | - | - | | **Section Header** | The heading shown above the download section on the partner's mailbox page. Defaults to **Files For You**. | ##### Receive Folders | Setting | Description | | - | - | | **Section Header** | The heading shown above the upload section on the partner's mailbox page. Defaults to **Your submissions**. | ### Settings Tab #### Connector Info *General settings for identifying and describing the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | ### Advanced Tab #### Advanced Settings | Setting | Description | | - | - | | **Upload History Retention** | The number of days a partner upload remains visible in the submission history on their mailbox page. The default is 30 days. The file itself is passed to the flow the moment it is accepted; this setting controls how long the partner can see that they submitted it. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## How Routing Works Routing in the Partner Mailbox connector uses the standard `SubFolder` header mechanism used by other MFT connectors. **Partner uploads (Receive folders)** When a partner uploads a file, it is written to the Receive subfolder the partner selected on the mailbox page. The resulting message carries a `SubFolder` header set to that folder's path. A downstream connector such as [Branch](../connectors/branch) can then route the message based on that header value. This is the primary reason for defining multiple Receive subfolders: partners can indicate the type of file they are sending by choosing the appropriate folder, and the flow branches accordingly. **Staging files for a partner (Send folders)** To deliver a file to a specific Send subfolder on the partner's page, the upstream connector or script must set the `SubFolder` header to the desired subfolder path *before* the message reaches the Partner Mailbox connector. There is no routing UI in the connector itself. You must set the header earlier in the flow, typically by using a [Script](script) connector. > [!NOTE] > > Pay close attention when setting the `SubFolder` header in upstream connectors. If the header is missing or set to a path that does not exist, files are reported as undeliverable and added to the errored-messages view. See [Managing the Folder Tree](#managing-the-folder-tree) for details. ## Managing the Folder Tree Renaming, moving, or deleting a Send or Receive folder can affect files that are already staged or in flight. Before confirming any such change, Arc displays the number of files currently in the affected folder. Verify that no files will be stranded before proceeding. When a folder is renamed or moved, files already staged under the old path are not automatically updated. Any upstream connector or script that sets the `SubFolder` header to the old path must also be updated to use the new path. Files that are reported as undeliverable appear in the errored-messages view. You can requeue them once the missing folder is added or the upstream script is corrected. ## Retention and File Lifecycle The following rules govern how files in Send folders are managed after they are staged: * **Delete after download**: the file is removed from the partner's mailbox page after the first download. Setting **File Retention For Downloaded Files** to 0 has the same effect. * **Keep until retention passes**: the partner can re-download the file until the retention period expires. Each download is recorded as a successful send in the connector's transaction log. * **Retention is counted from the first download**, not from when the file was staged. Re-downloading does not reset the retention period. * **Files that are never downloaded are never purged** by the retention setting. They remain available until they are manually removed. * **Application and workspace-level Send folder [cleanup settings](../getting-started/administration/settings/cleanup-settings) do not affect Partner Mailbox Send folders.** The Partner Mailbox connector manages its retention independently. ## Partner Mailbox View Partners access the mailbox by navigating to the **Partner Mailbox URL** displayed on the Mailbox Config tab and entering their credentials. Authentication uses browser-native basic authentication. The mailbox page has two sections, as shown in the following image: ![The partner mailbox page showing the Files For You and Your submissions sections side by side](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/partner-mailbox-page.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=f16fa76e77de81a157c3079c18f57d49) ### Files For You (Send folders) Lists files staged for the partner to download. Each row shows the filename, file size, the date the file was staged, and status. Status is **Available** for files the partner has not yet downloaded, or **Downloaded** with the date of the first download. A download button appears on available rows only. Downloading an entire folder returns a zip file named after the folder. Files whose retention period has passed disappear from the list silently. ### Your submissions (Receive folders) Lists files the partner has uploaded. Each row shows the filename, file size, submitted date and time, and status. To upload a file, the partner drags and drops it onto the upload zone or uses the file picker, targeting the currently selected Receive subfolder. Successes and failures are shown as toast notifications; uploads blocked by the file extension filter include a notification explaining the rejection. # PDF Label Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/pdf-label Configuration and usage guide for the CData Arc PDF Label connector. The PDF connector supports generating rich visual business forms as PDF documents from XML, by using a PDF template. ## Key Capabilities * PDF form generation from XML input using template-based field mapping * Multi-page output support with selective page generation and barcode field integration * Support for Code39, Code128, GS1-128, and QR code barcode types with advanced configuration ## Overview PDF connectors translate XML input into PDF output. XML element names are matched with form field names in the PDF template. The value of each XML element is pushed into the appropriate form field to generate the PDF output file. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Template File** | The PDF file to use as an output template. This file must have form fields whose names match the incoming XML data. The fields must also be arranged in the format you want for all PDF files generated by the connector. The [Example](#example) below shows how the XML and the template file work together to generate output. | #### Other Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Generating a PDF Document The PDF connector reads XML input files and uses the data from these XML files to generate PDF documents that match the **Template File** PDF document specified on the [Settings tab](#settings-tab). The structure of these XML input files should match the structure described below, or the [legacy structure](#legacy-xml-format) described in the next section. ### XML Format Here is an example of an XML input file: ```xml theme={null} Some data >5789337490548932 ``` As shown in the example above, the XML should match this structure: * An *Items* element at the root * A *PDF* child element representing a PDF document * Inside the *PDF* element, an element that matches the name of each form field that should be dynamically populated with data The names for each form field (the names to use for elements in the *PDF* element) come from the **Template File** PDF document. Whatever value is used to label the form fields in this template should be used as the element names to ensure the correct data is injected into the appropriate field in the resulting PDF. #### Multi-Page Outputs If the template file PDF contains multiple pages, the connector automatically generates the appropriate number of pages when processing an XML input file. In addition, multiple copies of the template file can be generated from a single XML file, in which case the additional copies are appended as multiple PDF pages for the same single output file. To generate multiple copies of the template file as a multi-page output, simply repeat the *PDF* element that is a direct child of the root *Items* element. Each *PDF* element at this position is considered a separate instance of the template file, so it should contain all of the named form field values necessary to fill out the template. #### Output a Subset of Pages If the template file PDF contains multiple pages, the connector supports generating a subset of these template pages by specifying the appropriate pages in the XML input file. The *Items* element at the root of the input document accepts a *pages* attribute with a comma-delimited list of the page numbers (zero-indexed) that should be included in the output. For example, if the template file contains 4 pages, and only the first and last page should be included in the output, the XML input file would look like this: ```xml theme={null} Some data ``` ### Barcodes Barcode fields (form fields that should contain barcode data) are treated just like other form fields: the barcode field has a label in the PDF template which should be used as the element name in the input XML. In addition, barcode fields should have a *type* attribute that determines the barcode type to generate. The connector supports four barcode types: * code39 * code128 * gs1128 * qrcode The numerical data for the barcode is then provided as the XML value for the appropriately named element. #### Advanced Barcode Configuration In addition to specifying the barcode type in the XML input file, the barcode can be configured directly in the PDF template file. When creating and editing the form field that holds the barcode, you can add configuration information to the field tooltip that affects the resulting bar code. This configuration information should always have the following form: `$$parameter1:value,parameter2:value$$` The following parameters are supported for barcodes: * **barcode**: the type of barcode (redundant when using the *type* attribute described in the previous section) * **qrcodeversion**: the version of the QR code when the barcode type is set to *qrcode* * **eclevel**: the level of error correction when using a QR code. Accepted values are *L* (low), *M* (medium), *Q* (quartile), and *H* (high) * **quietzone**: a whitespace buffer surrounding the barcode. The value should be an integer value representing a multiple of the smallest bar size (with a recommended value 10 or higher) The following parameters are uncommon and should only be used for barcodes with special requirements: * **heightnarrowratio**: the width of the narrowest bar in relation to the height of the barcode; must be specified as a ratio where height is 1 (for example, '0.05' for 1/20th) * **widenarrowratio**: the width of the narrowest bar in relation to the width of the widest bar; must be specified as a ratio where the widest bar is 1 (for example, '0.2' for 1/5th) * **spacebarratio**: the width of a space in a barcode in relation to the width of the bar; must be specified as a ratio where the bar width is 1 (default value is 1) ## Example Below is an example XML input file, PDF **Template File**, and resulting PDF output. > [!NOTE] > > The template file image below is of the Adobe Pro form editor, which displays the names of each form field. Remember that the field names in the template file must match the element labels in the XML. ### Input XML ```xml theme={null} WAREHOUSE 5B CLEARING HOUSE 14C 5789337490548932 5789337490548932 Fedex 12345 54321 87E112CA EE12FUZZY 4 99123445215586 99123445215586 1 1 8821445 2709667490538981 2709667490538981 ``` ### Template File ![PDF Template](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/label_1010_screenshot.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=c36429369c50ebe6d175116f17e18d95) ### Output ![PDF Sample](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/label_1010_filled.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=be4aae07dd0906accc3e48e4cfcaed71) ## Legacy XML Format In addition to the XML format described in [Generating a PDF Document](#generating-a-pdf-document), the connector supports an older XML input file structure to maintain backwards compatibility. CData recommends that you use the syntax described in the previous section when designing new Flows. Following is an example of the legacy XML format: ```xml theme={null} Wayne Tech Plant 1 100 Wayne Ave. Chapel Hill, NC 27514 Same as billing address ``` In this structure, all children of the root *Items* element should be called *Item*, and the *name* XML attribute should match the form field name in the PDF template file. The same XML attributes described in the previous sections are available in the legacy format (such as *type* for specifying barcode fields, and *pages* for generating a subset of output pages). ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Peppol Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/peppol Configuration and usage guide for the CData Arc Peppol connector, which enables secure automated exchange of e-invoices and business documents over the Peppol network. The Peppol connector enables secure, automated exchange of e-invoices and other business documents over the Peppol (Pan-European Public Procurement Online) network. This connector allows organizations to integrate their systems with the Peppol eDelivery infrastructure directly from within Arc, streamlining compliance with electronic procurement and invoicing mandates across Europe and beyond. The network facilitates the transmission of critical business documents through standardized Universal Business Language (UBL) formats and secure [AS4](./as4) protocol messaging. The Peppol network operates on a four-corner model architecture consisting of the sender, the sender's Access Point (AP), the receiver's AP, and the final receiver. This distributed infrastructure ensures reliable and standardized communication pathways for electronic procurement processes across international boundaries, with Access Points serving as certified gateways that handle the secure transmission of documents between participants on the network. ## Key Capabilities * Pan-European e-invoicing and e-procurement network integration using AS4-based four-corner model architecture * Universal Business Language (UBL) format support with automatic message routing based on party identifiers * SMP integration for dynamic endpoint discovery with certificate-based encryption and digital signatures ## Overview A Peppol connection is configured in two places. Configure the Peppol [Profiles](../getting-started/administration/profiles) page with a local party identifier and a private certificate. Then configure individual Peppol connectors with connection settings specific to a single trading partner. When an input file is processed by a Peppol connector, it is packaged and sent to the specified trading partner. When Arc receives a file over Peppol, it attempts to route the file to a specific Peppol connector. The application uses the Peppol party identifiers in the Peppol message to determine which Peppol connector should receive the file. When a file is routed to a Peppol connector, that file is placed in the connector's Transactions tab, and is passed along to the next connected connector in the flow (when there is one). ### Prerequisites Before you can configure and use the Peppol connector, you must complete the following: 1. Go to [https://Peppol.org/](https://Peppol.org/) to register and complete the certification to become a Peppol access point. 2. Register with an external SMP (Service Metadata Publisher). You must either register with an existing SMP or host your own. ## Profile Configuration The Peppol Profile must be configured before connections can be established with individual Peppol connectors. Click **Profiles** on the navbar, then click the **Peppol** tab. ### Peppol Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **Party Identifier** | The Party Id used by Arc in AS4 Peppol exchanges. Identifiers are case-sensitive. | #### Personal Certificate *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in a Peppol transaction: a corresponding public key is also generated with the same filename and a `.cer` extension. | | **Private Certificate Password** | The password required to access the **Private Certificate**. | #### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Receiving URL** | The URL at which Arc listens for incoming Peppol messages. Share this URL with all your trading partners. It is in this format: `/pub/Peppol.rsb`. | ## Connector Configuration Once you configure the global Peppol profile settings, create and configure individual Peppol connectors for each trading partner on the [Flows](../flows/flows) page. ### Settings Tab #### Configuration | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Peppol ID** | The unique identifier of the trading partner you are exchanging documents with. This is how you connect to the Peppol network, which enables secure and targeted message exchange. | #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Duplicate Message Action** | How the connector should behave when it receives a Peppol message with a Message Id that the connector has seen before. The connector detects duplicate messages by remembering Message Ids that have been received within the number of minutes specified in **Duplicate Interval**. | | **Duplicate Interval** | The number of minutes that a message with the same Message Id is considered a duplicate. If you set this to 0, the Message Ids are stored until the server is restarted. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as Peppol messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connector retries the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. ## Sending Documents When sending documents, the following occurs: * The connector uses the **Peppol Id** to look up the metadata (such as the endpoint and certificates) for the partner that has that ID. * The receiver defined in the Standard Business Document Header (SBDH) of the outbound message must match the **Peppol Id** of the connector where the message being sent. * You can configure the SBDH using an [XML Map](./xml-map/xml-map) connector when it is connected as the input to a Peppol connector. * The metadata that is returned from the lookup is used by the connector to send the document to the partner. ## Receiving Documents When receiving documents, the steps described in [Sending](#sending-documents) happen in reverse: * Your partner's Peppol AP looks up the metadata details that you provided to the SMP you registered with. * That metadata is used by your partner's Peppol AP to send you the message(s). When a message reaches Arc, the engine looks at the sender **Peppol Id** defined in the message SBDH, then routes the message to the Peppol connector that is configured with that **Peppol Id**. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# PIP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/pip Configuration and usage guide for the CData Arc PIP connector, which supports generating and processing RosettaNet Partner Interface Process (PIP) service headers for B2B messaging. The PIP connector supports generating RosettaNet Partner Interface Processes (PIP) service headers for outbound messages and processing service headers on inbound PIP messages. This connector should be used with the [RosettaNet](./rnif) connector (RNIF). ## Key Capabilities * RosettaNet PIP service header generation and validation for B2B messaging * Integration with RNIF for complete messaging workflows with configurable PIP codes, versions, and business activities * DTD-based schema validation for strict PIP specification compliance ## Overview When receiving a PIP message over RNIF, a PIP connector validates the PIP service headers against the settings configured for the connector. These settings include the PIP Code, PIP Version, and so on. After validation, the PIP message payload is processed into the **Transactions** tab for this connector, and passed along to the next connector in the flow. When generating PIP messages, a PIP connector adds service headers from the settings configured for the connector. Once these service headers are applied, the resulting file in the **Transactions** tab can be passed to a RosettaNet connector for transport to the trading partner. PIP connectors do not generate acknowledgments. Acknowledgments are handled by the RosettaNet connector. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Connector Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Direction** | Whether the connector is handling PIP messages that were received or PIP messages that need to be sent. | #### PIP Settings *Settings related to the service headers of PIP messages. When sending messages, these settings are applied as service headers in the resulting message. When receiving messages, these settings are used to validate the incoming document.* | Setting | Description | | - | - | | **PIP Code** | Identifies the standard business process associated with this message. | | **PIP Version** | Identifies the version of the PIP standard. | | **Business Activity Identifier** | Identifies a specific business activity. This code is found in table 3.2 of a PIP specification document. | | **Global Business Action Code** | Identifies a specific business action. This code is found in table 4.2 of a PIP specification document. | | **From Role** | The role initiating a business document exchange. | | **From Service** | The service from which a message is being sent. | | **To Role** | The role receiving the document in a business document exchange. | | **To Service** | The service to which a message is being sent. | | **Test Indicator** | Whether the message exchange is in test mode or production mode. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should validate the PIP against the DTD file and perform additional schema validation checks. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# PSV Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/psv The PSV connector converts Pipe-Separated Values files into XML and generates PSV files from XML, with configurable delimiters and encoding support. The PSV connector converts Pipe-Separated Values (PSV) files into XML and generates PSV files from XML. ## Key Capabilities * Bidirectional PSV/XML conversion with configurable delimiters and encoding * Header detection and custom column mapping capabilities * Support for various character encodings and special character handling ## Overview XML is the primary format that CData Arc uses to manipulate data within a flow. Therefore, it is useful to convert PSV files into XML as a staging step for further processing in the flow, or to convert XML to a PSV file after the XML has been manipulated. Both of these operations can be accomplished with the PSV connector. For details on converting PSV into XML, see the [Converting PSV to XML](#converting-psv-to-xml) section. For details on converting XML into PSV, see the [Converting XML to PSV](#converting-xml-to-psv) section. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the connector Id and description.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Connector Settings *Settings related to the core operation of the connector.*
SettingDescription
**Column headers present** Whether the PSV file contains a row of headers providing names or context to the values in the file.
**Record Name** The name of elements representing a row in the PSV file when converting to XML. You can use the following macros: %ConnectorID%, %FilenameNoExt%, %RegexFilename:%, and %Header:%. See [Converting PSV to XML](#converting-psv-to-xml) for more details.
#### Other Settings *Settings not included in the previous categories.*
SettingDescription
**Local File Scheme** A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros).
**Line Separator** The character sequence to use to denote line endings. By default, this is determined by the system, but you can use this setting to override the system behavior. Use `CR` or `\r` to denote carriage return, and `LF` or `\n` to denote line feed.
### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**.
  1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**.
  2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**.
  3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**.
### Advanced Tab
SettingDescription
**Processing Delay** The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.*
SettingDescription
**Send** Whether messages arriving at the connector are automatically processed.
#### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting PSV to XML When a PSV file is transformed into XML, the resulting XML has the following structure: ```xml theme={null} ``` Each row (record) in the original file becomes a child of the root element `Items`. The name of all record elements is determined by the **Record Name** option set in [Connector Settings](#connector-settings). Each record element then has child elements corresponding to the values in each row of the input file. Some PSV files include a line of header information that provides context to the values in the file. When the **First line is header information** setting is enabled, this header line is parsed, and the parsed headers are used as the element names for the value elements (the children of the record elements). Otherwise, the value elements are given generic names such as `field_0`, `field_1`, and so on. ## Converting XML to PSV To convert XML to a PSV file, the input XML must have a 'flat' structure. This means that, disregarding the `Items` root element, the depth of the XML structure is two. For example: ```xml theme={null} Citizen Kane 1941 119 Sharknado 2013 86 ``` The XML is interpreted as follows: * Children of the root element are treated as records (rows) in the resulting file * Children of each record element are treated as the values in each row If the **First line is header information** option is enabled, a header row is inserted into the resulting PSV file with the names of each value element to provide context to the values. In the example above, this header row would consist of `title`, `year`, and `runtime`. ## Print View When you view a PSV document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![](/26.4/images/psv_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## PSV Transformation: Using the XML Map Connector Many data transformation flows use the PSV connector in conjunction with the [XML Map Connector](/26.4/self-hosted/en/connectors/xml-map/xml-map). Often, data enters an Arc flow in PSV format and should exit the flow in some other format (for example, a database insert, an EDI file, or an insert into a CRM or ERP data source), or vice versa. Arc uses a single streamlined approach to these data transformation requirements:
  • Model the input format as XML
  • Model the output format as XML
  • Use the XML Map connector to map between the input XML and the output XML
Therefore, the PSV connector is commonly adjacent to an XML Map connector in the flow:
  • When PSV files are the *input* to the flow, the PSV connector converts a PSV file to XML and then passes that XML off to the XML Map connector to be transformed
  • When PSV files are the *output* from the flow, the PSV connector receives XML from the XML Map connector and converts it into a PSV file
An XML Map connector requires a sample XML structure for both the mapping input/source and the mapping output/destination. Use the [Sample Files](#sample-files-tab) tab to upload a sample file. Next, connect a PSV connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the PSV connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > The structure of this test file should be representative of future files. In other words, all of the PSV files processed by the PSV connector (and then the XML Map connector) should have the same columns as your test file. You might need to set up multiple PSV connectors and multiple XML Map connectors to handle distinct PSV structures. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Python Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/python Perform custom functions and operations using Python script, including data transformations, external calls, and business rule processing. The Python connector provides the ability to perform functions or operations that are not otherwise available in the built-in connectors of CData Arc. ## Key Capabilities * Custom business logic implementation using [Python scripting](../scripting/python-scripting) * Access to full message context including message headers and file content * Integration with external libraries and system commands (Python) * Support for complex data transformations and business rule processing ## Overview Python connectors use the Python scripting language to accomplish a wide range of custom behavior. The script configured in the connector is run each time a document is placed in the [Transactions](#transactions-tab) tab for the connector. The input document can be read and modified in the script, or the input document can simply trigger the Python script configured in the connector. Python connectors can also be configured to run on a schedule via **Receive** automation on the [Automation](#automation-tab) tab. After you decide the appropriate task for your Python connector, read the [Writing Python](../scripting/writing-python) documentation to explore how to call various operations. ## Connector Configuration This section contains all of the configurable connector properties. ### Python Tab *Options related to writing and testing the script.* ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/python_tab.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=c3bdb7e47a300003c0a029df64572bce) | Setting | Description | | - | - | | **Action dropdown** | Choose the connector's action. (See [Actions](#actions) for details on the three options.) | | **Python editor** | Use the editor to write your script. The toolbar contains standard editing options such as undo, redo, delete, copy, and format. It also includes an option to hide the Insert Options in the right pane. | | **Insert Options** | The panel in the right pane makes it easy to insert code into your script with one click. The dropdown includes **Inputs and Outputs** (such as ConnectorId or Encoding—the available options depend on the configured Action), **Snippets**, Keywords, Operators and Formatters. Find the code you want to insert, and click to add it. | | **Testing tab** | Specify whether to view the script input and/or output as the message body or headers. Options on this tab vary based on the script's action type (for example, trigger scripts do not have inputs). Click **Test** to test the script. | | **Log tab** | Displays the test log. | | **Last update on** | Displays the datetime the script was last modified and saved. | | **Auto update** | Use the toggle to automatically test the script after each modification. | #### Actions This connector can perform any of the three connector action types: • *Trigger* executes a script on a schedule and might produce files to be sent down the flow. This action serves as the starting point of a flow. • *Transform* accepts messages from the flow as input for a script and produces output. This action serves as the middle of a flow. • *Terminal* accepts messages from the flow as input for a script and acts as the terminal point of the flow. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | | **Retry Interval** | The number of minutes that the connector waits before retrying a failed script execution. | | **Max Attempts** | The number of times that the connector retries a failed script execution before raising an error. | | **Receive** | Whether the connector should automatically run the script on a configured schedule. | | **Receive Execution Interval** | The unit of measure to use for the receive automation schedule (such as Minutes, Hours, or Days). | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Options on this tab vary depending on the connector's [action type](#actions): * If your connector is a **Trigger**, use the **Receive** button to start the flow. * If your connector is a **Transform** or **Terminal**, messages that come from a previous connector in the flow automatically appear in the flow, or you can use the **Upload Files** button to add files. ## Producing Output and Log Entries Python connectors send data to the output feed through the built-in [`output`](../scripting/writing-python#output) item, which is pushed automatically when the script finishes — no explicit push is needed. (To push a custom item instead, set its attributes and call [`_ctx.push(item)`](../scripting/writing-python#push).) Following are three common use cases. 1 Create and output a file: ```xml theme={null} output.Filename = "results.txt" output.Data = "This is my result" ``` 2 Pass an input file as output: ```xml theme={null} output.Filepath = _input.Filepath ``` 3 Write entries to the connector log file: ```xml theme={null} print("Custom Log entry line 1") print("Custom Log entry line 2") ``` In a Python connector, `print()` writes to the log file for the current message (as shown above). To write to the Arc [application log](../getting-started/administration/activity#application-logs) instead, use `_ctx.log('level', 'message')` with a level of `DEBUG`, `INFO`, `WARNING`, or `ERROR`. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# REST Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/rest Build dynamic REST requests to consume RESTful API web services, with support for all HTTP methods, OAuth 2.0, and ArcScript-driven URLs and headers. REST connectors support building dynamic REST requests to consume RESTful API web services. ## Key Capabilities * Complete RESTful API client with support for all HTTP methods (GET, POST, PUT, PATCH, and DELETE) * Swagger import capability for automatic API configuration and request generation * Advanced authentication including OAuth 2.0, Bearer Token, AWS Signature, API Key and Basic/Digest auth * Dynamic request building with ArcScript support in URLs, headers, and form data * Flexible body types including raw, form-data, URL-encoded, and file uploads ## Overview REST connectors expose a simple interface to build the headers, authorization, body, and HTTP method for a REST request. The request body can be set statically in the connector configuration or dynamically generated based on the files processed by the connector. ### Actions This connector can perform any of the three connector action types: • *Trigger* executes a request on a schedule and might produce files to be sent down the flow. This action serves as the starting point of a flow. • *Transform* accepts messages from the flow as input for a request and produces output. This action serves as the middle of a flow. • *Terminal* accepts messages from the flow as input for a request and acts as the terminal point of the flow. ## Connector Configuration ### Settings Tab *Settings related to the core configuration of the connector.* #### Configuration
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### REST Details Tab *Settings related to the request details for the connector.* ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_details_tab.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=d5b83dcbe8cfe959fefdc8972b714b7f) | Setting | Description | | - | - | | **REST Details** | Select the action the connector should execute. (See [Actions](#actions) for details on the three options.) | | **Import from Swagger** | Click the ![import from swagger](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_import_from_swagger.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=b74144e153de876d68d5dd25b4dab539) icon to import API details directly from a Swagger URL. | | **Test** | Test the current configuration without creating a message or transaction that is sent down the flow. Clicking **Test** opens a window with additional tabs (see [Testing Request Configurations](#testing-request-configurations) for more information). | | **Request URL** | The HTTP method and target HTTP URL for the REST request. The available method options are: GET, POST, PUT, PATCH, and DELETE. | | **Allow ArcScript in URL** | Check this to allow the evaluation of ArcScript expressions in the URL before the query is issued. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_arcscript_editor.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=05d3576b0141d612fcc6e83469592e28)), which enables you to build dynamic request URLs using ArcScript. See [Using the ArcScript Editor for Request URLs and Headers](#using-the-arcscript-editor-for-request-urls-and-headers) for details. | | **Body Type** | The type of body content supplied with the REST request. Not visible when the HTTP method is GET. See [Body](#body) for descriptions of each option. | | **Content Type** | The type of content used for raw data transfers. Only visible when **Body Type** is set to `raw`. | #### Query Params Use this tab to add query parameters to your request URL. Supply the parameters as name-value pairs, with optional descriptions. Click **Save** to update the URL with your parameters. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_query_params.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=2b8d6ce30c17d2bd7cded2c0f31cbf86) The previous image includes two sets of query parameters. When the connector is saved, the `api.example.com` request URL becomes: `https://api.example.com/orders?status=pending&limit=50` Click the ellipses on the right to set two other options: **Bulk Edit View** and **Hide Description Column**. #### Authorization Use this tab to configure authentication credential for API requests. | Setting | Description | | - | - | | **Credentials** | Whether to supply authentication credentials from a previously configured Connection, create a new connection, or send requests without authentication. **From Connection**: Select this, then choose your **Connection** from the dropdown. **Note**: Create a new connection by clicking the plus sign next to **Connection**. **No Credentials**: Select this to send requests without authentication. Use this option for public APIs or when authentication is handled through [query parameters](#query-params) or [headers](#headers). | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS server. You can upload a certificate, leave the field blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to trust the target server's identity. Use Any Certificate with caution: the certificate verifies that you are connecting to the intended server. | #### Headers This tab enables you to add a list of HTTP headers to include in the outgoing REST request. Headers are also specified as name-value pairs. See [Static Requests](#static-requests) and [Dynamic Requests](#dynamic-requests) for more information. Click the ellipses on the right to access three other options: **Bulk Edit View**, **Allow ArcScript in Headers**, and **Show Autogenerated Headers**. See [Using the ArcScript Editor for Request URLs and Headers](#using-the-arcscript-editor-for-request-urls-and-headers) for details on using the ArcScript editor. #### Body If the **Body Type** is set to `form-data` or `x-www-urlencoded`, use the **Body** tab to supply the set of name-value pairs (fields) that form the body of the request. The following list describes each option in more detail. * **none**: No body is supplied with the REST request. * **form-data**: The body is supplied as a set of name-value pairs (fields). Use the dropdown list next to **Name** to select the field type. * **Static**: Supply both the **Name** and the **Value**. * **XML**: Supply the **Name** in the UI. The **Value** is dynamically read from the input file processed by the connector. For more information, see [Dynamic Form Data](#dynamic-form-data). * **File**: Each connector can have one body field set to File. This causes the input file to be sent in the body of the request. The **Value** field is greyed out since the connector uses the input file itself as the form data. * **Header**: Use the **Value** field to specify which header from the input message to read from for the body. * **ArcScript**: The **Value** supplied is rendered as ArcScript, and the resulting value is used in the request body. * **x-www-urlencoded**: The body is configured in the same way as form-data; however, the name-value pairs are encoded as a URL query string instead of multipart form data. * **raw**: The body is set to the contents of the input file processed by the connector. Select the **Content Type** of the body using the dropdown, or by specifying it as a custom header in the **Header** section. Click the ellipses on the right to access two other settings: **Bulk Edit View** and **Show Content Type Column**. Showing the content type column enables you to supply the content type on a per-field basis for `form-data` and `x-www-form-urlencoded` body types. #### Options *Additional settings related to the request.* | Setting | Description | | - | - | | **Create an output message for error responses** | By default, the connector does not output a message to the flow when the response is sent with a status code that does not indicate success. Enable this setting to have the connector output a message to the flow for these error responses. You can use the HTTP-Status-Code header to filter messages later in the flow. | | **Compress HTTP Request** | Enable this to compress the body of PUT or POST requests into `gzip` format before sending the request. The **Content-Encoding** header is also added to the outgoing request. | | **GET Request Body** | Allows GET requests to use the input message data as the body of the request. | | **HTTP Version** | Whether to use HTTP 1.0, 1.1, or 2.0 when connecting to the REST service. | ### Advanced Tab #### Authentication *Settings related to client authentication when two-way TLS authentication is required.*
SettingDescription
**Private Certificate** The private certificate presented during TLS client authentication.
**Certificate Password** The password required to access the TLS client certificate.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Chunked Encoding** | Whether to use HTTP Chunked Transfer Encoding when sending requests. This allows the application to send portions (chunks) of the message sequentially to avoid overloading the connection. | | **Chunk Size** | The size, in bytes, of each chunk when **Chunked Encoding** is enabled. | | **Follow Authorization Header** | When enabled, retains the authorization header when a redirect happens to a different protocol or hostname. | | **Output Behavior** | By default, the connector outputs a message containing the `Response` data. The other options allow you to output the `Input Message` so it can be processed further, or to not output a message at all. See [Response Event](#response-event) for more information. | | **Timeout (seconds)** | The duration in seconds to wait for a response from the REST server before throwing a timeout error. | | **Response Headers** | When set, the connector promotes the specified headers from the REST message as metadata on the downloaded message. You can specify multiple headers in a comma-separated list. | | **Preserve Cookies** | To preserve cookies on output messages, supply a comma-separated list of the cookie names to persist. Use `*` to preserve all cookies. **Note**: Only the name-value pairs are preserved, and no attribute checks are performed. Users are responsible for ensuring cookies are not leaked to unintended destinations. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | | **Receive** | A toggle that instructs the connector to automatically make the REST request at the configured interval. | | **Interval** | The interval at which the connector makes the configured REST request. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service sends requests at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before making the request. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Options on this tab vary depending on the connector's [action type](#actions): * If your connector is a **Trigger**, use the **Receive Files** button to start the flow. * If your connector is a **Transform** or **Terminal**, use the **Upload Files** button to upload files to the flow. ## Establishing a Connection A valid target URL is required to establish a connection to any REST service. The service URL can support various HTTP methods, and you should configure the method based on the particular web service action or dataset to retrieve. Some services might also require authentication or a set of custom headers in order to consume the service. The **Credentials** section of the [Authorization](#authorization) tab enables you to specify your connection credentials. Choose from the following options: * **From Connection**: Choose a previously configured Arc shared Connection, or click the plus sign next to the **Connection** field to [Create a Connection](#create-a-connection). * **No Credentials** for public APIs or when authentication is being handled through [query parameters](#query-params) or [headers](#headers). If the target URL is an HTTPS URL, set the **TLS Server Certificate** to the public key certificate identifying the server. To implicitly trust the target endpoint, set the field to `Any Certificate`. ### Create a Connection To create a new connection, choose **From Connection**, then click the plus sign next to the **Connection** field. * Enter a unique **Connection Name**. * The **Type** is always set to REST. * Choose your **Auth Scheme**. See [Authenticating](#authenticating) for details. ### Authenticating The REST connector supports a number of authentication types, and each type has its own requirements: * Basic (plain text), Digest (encrypted), and NTLM require username-password authentication. These credentials are supplied to the REST service as headers in the request. * OAuth authentication requires an app registration in the REST service's web portal or development console. The **Callback URL** to include in the app registration is shown in the Arc UI. Choose the **Grant Type** that applies to the REST service, and specify the remaining settings from the details shown in the REST service's web portal or development console. Then click **Get New Access Token** to obtain the necessary tokens to interact with the service. Once the initial tokens have been retrieved, the application refreshes the tokens as they approach expiration. * Bearer Token authentication requires a token from the service's web portal or development console. * AWS Signature authentication, for authenticating against Amazon, requires configuration credentials provided by Amazon: **Access Key**, **Secret Key**, and so forth. * API Key authentication requires a key-value pair, then you must specify whether the key should be added as a [header](#headers) or as a [query parameter](#query-params). ## Testing Request Configurations At any time you can test your current configuration without creating a message or transaction that is sent down the flow. Click **Test** on the **REST Details** tab. The following image shows the **Response Body** results of a successful test of a Trigger connector. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_test_tab.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=3035ccb5e8f0f79e517aaf2749486784) * **Response Body**: Displays the output of the REST request in the format that was returned from the server. * **Response Headers**: Displays the response headers that were included in the response returned from the server. * **Message Headers**: Displays the message headers that were included in the output of the test. * **Log**: Displays the test logs. Transform and Terminal connectors have an **Input** pane with tabs for **XML**, **Headers**, and **Logs**. * The **XML** tab is only populated if you have defined XML fields in the body of your request, as shown in the following image. * If **Allow ArcScript in URL** or **Allow ArcScript in Headers** is checked, use the **Headers** tab to supply message headers that might be used in scripting contexts (see [Using the ArcScript Editor for Request URLs and Headers](#using-the-arcscript-editor-for-request-urls-and-headers) for details on those options). If your request has a [Header](#headers) configured as an element of the `form-data` or `x-www-form-urlencoded` body, the header name appears here and you can supply a value to use for the test. * The **Logs** tab contains the results of your last test. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_test_tab_transform.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=d88b989b4a8cd63a5504b8450fd18836) ## Static Requests REST requests that have entirely static content (such as requests that use the HTTP GET method) do not require an input file, since the request content is configured entirely in the connector UI. Simply add any necessary name-value pairs as custom headers in the **Header** section or form data in the **Body** section. Static requests can be automatically sent according to a schedule if **Receive Automation** is enabled. The response to each request is stored in the output folder or passed along to the next connector in the flow. If **Send Automation** is enabled, files arriving at the connector **Transactions** folder also trigger a static request. The content of the input file is ignored, and the request is sent according to the configuration in the UI. ## Dynamic Requests REST requests can be dynamically populated with data from files that arrive in the connector **Transactions** folder. ### Raw Input Data If you set the **Body Type** of the request to `raw`, the content of input files is sent as the body of the REST request. Use the **Content Type** dropdown to set the specific content type of the data. If the content type you need is not listed, you can add a *Content-Type* header in the **Header** section. ### Dynamic Form Data If you set the **Body Type** of the request to `form-data` or `x-www-urlencoded`, the connector looks for specific values from the input file to populate the request. For each name-value pair that is set to `XML`, the connector scans input files for an XML element that shares the same name as the field name, and that uses a specific XML structure, as shown below: ```xml theme={null} ``` To fit this structure, CData strongly recommends that you use an XML Map connector in front of the REST connector in the flow, as described [below](#dynamic-templates-with-xml-map). When the connector finds an element that matches the field name and required XML structure, the value in this element is used as the value in the name-value pair. For example, if the body has a dynamic field with the name `CustomerID`, and the input file has the XML shown below, the REST connector sets the value of the `CustomerID` field to 12354. ```xml theme={null} 12354 ``` #### Dynamic Templates with XML Map Use the [XML Map](./xml-map/xml-map) connector in conjunction with the REST connector to easily build dynamic requests out of other XML data structures. The XML Map connector converts custom XML structures into the XML structure that the REST connector expects. First, configure the REST connector with the set of dynamic (and static) **Body** fields that should be present in the request. Next, connect an XML Map connector to the REST connector in the Arc flow and save the flow changes. This allows the XML Map connector to detect which fields the REST connector expects in incoming input files. Then, inside the XML Map connector, the **Destination File** dropdown includes the REST request schema. Select this as the Destination, and set the **Source File** to a custom XML structure. This populates the XML Map Mapping Editor, and you can drag and drop the data that needs to be included in the REST request from the source structure into the destination structure. Once the mapping is complete, the XML Map connector automatically converts files that match the source file into a valid REST request structure. For more information on using the XML Map connector, please see the [XML Map connector documentation](./xml-map/xml-map). #### Dynamic Headers Expressions in ArcScript can also be evaluated to generate dynamic strings as header values. See [Headers](#headers) for details and examples. ## Using the ArcScript Editor for Request URLs and Headers You can use the ArcScript editor shown below to build your request URLs and in headers. The following image shows the request URL editor, but the header value editor works the same way. ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_arcscript_editor_pane.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=081c75c0820f62bec10d66fe61180bed) Select **Allow ArcScript in URL** on the [REST Details tab](#rest-details-tab) of the connector configuration pane to allow expressions in ArcScript to be evaluated to generate dynamic strings as URLs. For example, the following URL includes the date and time: `http://myendpoint.com/api?day=[_ | now('yyyyMMdd HH:mm:ss')]` This URL includes a header on the incoming message for queries triggered through send automation: `http://myendpoint.com/api?customer=[_message.header:customerid]` Finally, this URL uses a dynamic date range beginning from the time of the last query to the current timestamp, using a default timestamp for the first query: `http://myendpoint.com/api?DateFrom=[_connector.lastruntimestamp | def('2025-01-01T00:00:00-04:00')]&DateTo=[_connector.currenttimestamp]` You can add the expressions directly to the URL, or use the editor to write them. Select **Allow ArcScript in Headers** to enable the evaluation of ArcScript expressions in [headers](#headers) before the query is issued. For example, the following header contains the date and time: `Timestamp [_ | now('yyyyMMdd')]` This header contains a customer Id for queries triggered through send automation: `Customer [_message.header:customerid]` ### Message Headers Message headers help Arc track the progress of data through flows. All Tracked Headers appear on the **Message Header** tab in the editor, and you can reference them in your expression. You can also include other message headers in your expression by using the **Add Message Header** field in the editor and providing the name of an existing header. These do not have to be tracked headers. ### Vault Use the **Vault** tab to add items from your [Global Settings Vault](../getting-started/administration/settings/global-settings-vault) to an expression. This is useful if you have values that you reuse in different places throughout a flow. You can define those values in the vault, then reference them at the beginning of the expression. Keep in mind that if you want the mapping to use the *value* of the item in the vault, you need to reference it inside square brackets; otherwise the editor interprets the item *name* as a literal. ### Formatters Formatters support manipulating the values returned at different xpaths. Formatters are separated by a pipe character (|) in the expression, and evaluated from left to right. For example: `[xpath('City') | toupper | substring(0,3)]` In this example, before the value of the `City` xpath is returned, all string characters are converted to upper case characters, and a substring of the first three characters is returned in the result. For example, if the source document had a value of: `Durham` The resulting expression returns the following: `DUR` The formatters are listed on the **Formatters** tab. Add a formatter to the expression by clicking it in the list. ## Response Event You can use a response event to interact with the response received from the server (which includes the body, headers, cookies, and so on), and to enrich the output message generated from the connector. You can use the following special items in the `Response` event. * [\_cookies](../scripting/introduction-to-arcscript#http-cookies-_cookies) * [\_message](../scripting/introduction-to-arcscript#message-_message) * [\_request](../scripting/introduction-to-arcscript#http-request-_request) * [\_response](../scripting/introduction-to-arcscript#http-response-_response) ### Response Event Example This script reads the JSON response received from the server, parses out the access token contained in the JSON, and adds it as a header onto the output message created by the connector: ```xml theme={null} ``` The following steps detail what happens: 1. The body of the response sent by the server back to the connector is accessed via `_response.body` and is set to the text attribute of the jsonDOMGet operation. Other attributes of jsonDOMGet are also populated, such as the `map` attribute with the jsonpath to the desired token in the response body. 2. The jsonDOMGet operation is called. If the token is found in the response JSON body, it is added as a message header to the output message of the connector via the `_message.header:access_token` syntax. If the token is not found, the value of the `access_token` header is set to a static string: *Token not found!*. Here is what that result looks like in Arc when you view the output message details of the message from the connector: ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_response_event.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=bf069730a3bde367d12c37d0342f298c) This type of script is useful when you need to parse data from the raw JSON response body received back from a server after a request is sent to it. The header can then be read and used in subsequent connectors in the flow. > [!TIP] > > If the server responds using XML, you can achieve the same result using xmlDOMGet. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# REST V1 Connector (Deprecated) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/restv1 The original REST connector for building dynamic REST requests. This version is deprecated; use the current REST connector for new flows. REST connectors support building dynamic REST requests to consume RESTful API web services. ## Key Capabilities * Complete RESTful API client with support for all HTTP methods (GET, POST, PUT, PATCH, and DELETE) * Swagger import capability for automatic API configuration and request generation * Advanced authentication including OAuth 2.0, Bearer Token, AWS Signature, and Basic/Digest auth * Dynamic request building with ArcScript support in URLs, headers, and form data * Flexible body types including raw, form-data, URL-encoded, and file uploads ## Overview REST connectors expose a simple interface to build the headers, authorization, body, and HTTP method for a REST request. The request body can be set statically in the connector configuration or dynamically generated based on the files processed by the connector. ### Actions This connector can perform any of the three connector action types: • *Trigger* executes a request on a schedule and might produce files to be sent down the flow. This action serves as the starting point of a flow. • *Transform* accepts messages from the flow as input for a request and produces output. This action serves as the middle of a flow. • *Terminal* accepts messages from the flow as input for a request and acts as the terminal point of the flow. ## Connector Configuration ### REST Details Tab *Settings related to the request details for the connector.* ![](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_details_tab.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=d5b83dcbe8cfe959fefdc8972b714b7f) #### Authentication *Settings related to authenticating with the REST service.* | Setting | Description | | - | - | | **Authentication Type** | The type of authentication to use with the REST service. Options vary based on your selection. See [Authenticating](#authenticating) for details. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS server. You can upload a certificate, leave the field blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to trust the target server's identity. Use Any Certificate with caution: the certificate verifies that you are connecting to the intended server. | #### Request Details | Setting | Description | | - | - | | **Import from Swagger** | Click the ![import from swagger](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_import_from_swagger.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=b74144e153de876d68d5dd25b4dab539) icon to import API details directly from a Swagger URL. | | **Test** | Test the current configuration without creating a message or transaction that is sent down the flow. Clicking **Test** opens a window with four tabs (see [Testing Request Configurations](#testing-request-configurations) for more information): **Response Body**: View the body of the response from the server. **Response Headers**: View the headers that were included in the response from the server. **Message Headers**: View the message headers that were included in the output of the test. **Log**: View the connector log. | | **Action** | Which action the connector should execute. (See [Actions](#actions) for details on the three options.) | | **Request URL** | The HTTP method and target HTTP URL for the REST request. The available method options are: GET, POST, PUT, PATCH, and DELETE. | | **Body Type** | The type of body content supplied with the REST request. Not visible when the HTTP method is GET. See [Body Types](#body-types) for descriptions of each option. | | **Headers** | The **Headers** tab supports adding a list of HTTP headers to include in the outgoing REST request. Headers are specified as name-value pairs. See [Static Requests](#static-requests) for more information. | | **Body** | If the **Body Type** is set to `form-data` or `x-www-urlencoded`, use the **Body** tab to supply a set of name-value pairs (fields) that form the body of the request. See [Body Types](#body-types) for details. | ### Settings Tab *Settings related to the core configuration of the connector.* #### Configuration
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.*
SettingDescription
**Private Certificate** The private certificate presented during TLS client authentication.
**Certificate Password** The password required to access the TLS client certificate.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **ArcScript in Headers** | Allows for the evaluation of ArcScript expressions in the headers before the query is issued. See the [URL](#url) section for examples. | | **ArcScript in URL** | Allows for the evaluation of ArcScript expressions in the URL before the query is issued. See the [Headers](#headers) section for examples. | | **Output Error Responses** | By default, the connector does not output a message to the flow when the response is sent with a status code that does not indicate success. Enable this setting to have the connector output a message to the flow for these error responses. You can use the HTTP-Status-Code header to filter messages later in the flow. | | **Chunked Encoding** | Whether to use HTTP Chunked Transfer Encoding when sending requests. This allows the application to send portions (chunks) of the message sequentially to avoid overloading the connection. | | **Chunk Size** | The size, in bytes, of each chunk when **Chunked Encoding** is enabled. | | **Compression** | Whether to compress the body of PUT or POST requests into `gzip` format before sending the request. The **Content-Encoding** header is also added to the outgoing request. | | **GET Request Body** | Allows GET requests to use the input message data as the body of the request. | | **HTTP Version** | Whether to use HTTP 1.0, 1.1, or 2.0 when connecting to the REST service. | | **Output Behavior** | By default, the connector outputs a message containing the `Response` data. The other options allow you to output the `Input Message` so it can be processed further, or to not output a message at all. See [Response Event](#response-event) for more information. | | **Timeout (seconds)** | The duration in seconds to wait for a response from the REST server before throwing a timeout error. | | **Response Headers** | When set, the connector promotes the specified headers from the REST message as metadata on the downloaded message. You can specify multiple headers in a comma-separated list. | | **Preserve Cookies** | To preserve cookies on output messages, supply a comma-separated list of the cookie names to persist. Use `*` to preserve all cookies. **Note**: Only the name-value pairs are preserved, and no attribute checks are performed. Users are responsible for ensuring cookies are not leaked to unintended destinations. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | | **Receive** | A toggle that instructs the connector to automatically make the REST request at the configured interval. | | **Receive Execution Interval** | The interval at which the connector makes the configured REST request. The next field depends on the selection here: **Hourly**: A *Minutes Past the Hour* dropdown menu allows you to specify the number of minutes past the hour to process receive files. **Daily**: A *Time* field appears to specify the time of day (in UTC) to process receive files. **Weekly**: Two fields appear. *Day* allows you to select the day of the week for processing, and *Time* allows you to specify the time (in UTC) to process receive files. **Monthly**: Two fields appear. *Day* allows you to select the day of the month for processing, and *Time* allows you to specify the time (in UTC) to process receive files. **Minute**: A *Minutes* field appears to specify the number of minutes between processing intervals. **Advanced**: A five-position *Cron Expression* field allows you to specify exact processing intervals. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when **Receive Interval** is set to *Hourly*. For example, if this value is set to 5, the automation service sends requests at 1:05, 2:05, 3:05, etc. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Options on this tab vary depending on the connector's [action type](#actions): * If your connector is a **Trigger**, use the **Receive Files** button to start the flow. * If your connector is a **Transform** or **Terminal**, use the **Upload Files** button to upload files to the flow. ## Establishing a Connection A valid target URL is required to establish a connection to any REST service. The service URL can support various HTTP methods, and you should configure the **Method** based on the particular web service action or data set to retrieve. Some services might also require authentication or a set of custom headers in order to consume the service. If the target URL is an HTTPS URL, set the **TLS Server Certificate** to the public key certificate identifying the server. To implicitly trust the target endpoint, set the field to `Any Certificate`. ### Authenticating The REST connector supports a number of authentication types, and each type has its own requirements: * Basic (plain text), Digest (encrypted), and NTLM require username-password authentication. These credentials are supplied to the REST service as headers in the request. * OAuth authentication requires an app registration in the REST service's web portal or development console. The **Callback URL** to include in the app registration is shown in the Arc UI. Choose the **Grant Type** that applies to the REST service, and specify the remaining settings from the details shown in the REST service's web portal or development console. Then click **Get New Access Token** to obtain the necessary tokens to interact with the service. Once the initial tokens have been retrieved, the application refreshes the tokens as they approach expiration. * Bearer Token authentication requires a token from the service's web portal or development console. * AWS Signature authentication, for authenticating against Amazon, requires configuration credentials provided by Amazon: **Access Key**, **Secret Key**, and so forth. ## Testing Request Configurations At any time you can test your current configuration without creating a message or transaction that is sent down the flow. Click **Test** on the **REST Details** tab. The following image shows the **Response Body** results of a successful test. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_test_tab.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=3035ccb5e8f0f79e517aaf2749486784) * **Response Body**: Displays the output of the REST request in the format that was returned from the server. * **Response Headers**: Displays the response headers that were included in the response returned from the server. * **Message Headers**: Displays the message headers that were included in the output of the test. * **Log**: Displays the test logs. ## Body Types Use the connector [Request Details](#request-details) tab to specify how the body is configured, and what type of information is supplied. The following list describes each option in more detail. * **none**: No body is supplied with the REST request. * **form-data**: The body is supplied as a set of name-value pairs (fields). Use the dropdown list next to **Name** to select the field type. * **Static**: Supply both the **Name** and the **Value**. * **XML**: Supply the **Name** in the UI. The **Value** is dynamically read from the input file processed by the connector. For more information, see [Dynamic Form Data](#dynamic-form-data). * **File**: Each connector can have one body field set to File. This causes the input file to be sent in the body of the request. The **Value** field is greyed out since the connector uses the input file itself as the form data.\ **Note**: The **File** option is incompatible with dynamic requests and other file requests. If you use this option, you can only combine it with static fields. * **Header**: Use the **Value** field to specify which header from the input message to read from for the body. * **ArcScript**: The **Value** supplied is rendered as ArcScript, and the resulting value is used in the request body. * **x-www-urlencoded**: The body is configured in the same way as form-data; however, the name-value pairs are encoded as a URL query string instead of multipart form data. * **raw**: The body is set to the contents of the input file processed by the connector. Select the **Content Type** of the body using the dropdown, or by specifying it as a custom header in the **Header** section. ## Static Requests REST requests that have entirely static content (such as requests that use the HTTP GET method) do not require an input file, since the request content is configured entirely in the connector UI. Simply add any necessary name-value pairs as custom headers in the **Header** section or form data in the **Body** section. Static requests can be automatically sent according to a schedule if **Receive Automation** is enabled. The response to each request is stored in the output folder or passed along to the next connector in the flow. If **Send Automation** is enabled, files arriving at the connector input folder also trigger a static request. The content of the input file is ignored, and the request is sent according to the configuration in the UI. ## Dynamic Requests REST requests can be dynamically populated with data from files that arrive in the connector input folder. ### Raw Input Data If you set the **Body Type** of the request to `raw`, the content of input files is sent as the body of the REST request. Use the **Content Type** dropdown to set the specific content type of the data. If the content type you need is not listed, you can add a *Content-Type* header in the **Header** section. ### Dynamic Form Data If you set the **Body Type** of the request to `form-data` or `x-www-urlencoded`, the connector looks for specific values from the input file to populate the request. For each name-value pair that is set to `XML`, the connector scans input files for an XML element that shares the same name as the field name, and that uses a specific XML structure, as shown below: ```xml theme={null} ``` To fit this structure, CData strongly recommends that you use an XML Map connector in front of the REST connector in the flow, as described [below](#dynamic-templates-with-xml-map). When the connector finds an element that matches the field name and required XML structure, the value in this element is used as the value in the name-value pair. For example, if the body has a dynamic field with the name `CustomerID`, and the input file has the XML shown below, the REST connector sets the value of the `CustomerID` field to 12354. ```xml theme={null} 12354 ``` #### Dynamic Templates with XML Map Use the [XML Map](./xml-map/xml-map) connector in conjunction with the REST connector to easily build dynamic requests out of other XML data structures. The XML Map connector converts custom XML structures into the XML structure that the REST connector expects. First, configure the REST connector with the set of dynamic (and static) **Body** fields that should be present in the request. Next, connect an XML Map connector to the REST connector in the Arc flow and save the flow changes. This allows the XML Map connector to detect which fields the REST connector expects in incoming input files. Then, inside the XML Map connector, the **Destination File** dropdown includes the REST request schema. Select this as the Destination, and set the **Source File** to a custom XML structure. This populates the XML Map Mapping Editor, and you can drag and drop the data that needs to be included in the REST request from the source structure into the destination structure. Once the mapping is complete, the XML Map connector automatically converts files that match the source file into a valid REST request structure. For more information on using the XML Map connector, please see the [XML Map connector documentation](./xml-map/xml-map). ### URL If you select **Allow ArcScript in URL** on the [Advanced tab](#advanced-tab) of the connector configuration pane, expressions in ArcScript can be evaluated to generate dynamic strings as URLs. For example, the following URL includes the date and time: `http://myendpoint.com/api?day=[_ | now('yyyyMMdd')]` This URL includes a header on the incoming message for queries triggered through send automation: `http://myendpoint.com/api?customer=[_message.header:customerid]` Finally, this URL uses a dynamic date range beginning from the time of the last query to the current timestamp, using a default timestamp for the first query: `http://myendpoint.com/api?DateFrom=[_connector.lastruntimestamp | def('2025-01-01T00:00:00-04:00')]&DateTo=[_connector.currenttimestamp]` ### Headers If you select **Allow ArcScript in Headers** on the [Advanced tab](#advanced-tab) of the connector configuration pane, expressions in ArcScript can be evaluated to generate dynamic strings as header values. For example, the following header contains the date and time: `Timestamp [_ | now('yyyyMMdd')]` The header below contains a customer Id for queries triggered through send automation: `Customer [_message.header:customerid]` ## Response Event You can use a response event in the REST connector to interact with the response received from the server (which includes the body, headers, cookies, and so on), and to enrich the output message generated from the connector. You can use the following special items in the `Response` event. * [\_cookies](../scripting/introduction-to-arcscript#http-cookies-_cookies) * [\_message](../scripting/introduction-to-arcscript#message-_message) * [\_request](../scripting/introduction-to-arcscript#http-request-_request) * [\_response](../scripting/introduction-to-arcscript#http-response-_response) ### Response Event Example This script reads the JSON response received from a REST call, parses out the access token contained in the JSON, and adds it as a header onto the output message created by the REST connector: ```xml theme={null} ``` The following steps detail what happens: 1 The body of the response sent by the server back to the REST connector in Arc is accessed via `_response.body` and is set to the text attribute of the jsonDOMGet operation. Other attributes of jsonDOMGet are also populated, such as the `map` attribute with the jsonpath to the desired token in the response body. 2 The jsonDOMGet operation is called. If the token is found in the response JSON body, it is added as a message header to the output message of the REST connector via the `_message.header:access_token` syntax. If the token is not found, the value of the `access_token` header is set to a static string: *Token not found!*. Here is what that result looks like in Arc when you view the output message details of the message from the REST connector: ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/rest_response_event.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=bf069730a3bde367d12c37d0342f298c) This type of script is useful when you need to parse data from the raw JSON response body received back from a server after a request is sent to it. The header can then be read and used in subsequent connectors in the flow. > [!TIP] > > If the server responds using XML, you can achieve the same result using xmlDOMget. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# RosettaNet Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/rnif Configuration and usage guide for the CData Arc RosettaNet connector. The RosettaNet connector sends and receives messages following the RosettaNet Implementation Framework (RNIF) standard. ## Key Capabilities * RNIF 1.1 and 2.0 support for B2B document exchange * Automatic message routing with per-partner configuration and flexible encryption/signing options * Certificate-based security with synchronous/asynchronous acknowledgment handling ## Overview A RosettaNet connection is configured in two places. Configure the RosettaNet [Profiles](../getting-started/administration/profiles) page with a local RosettaNet identifier, and private certificates. Then configure individual RosettaNet connectors with connection settings specific to a single trading partner. When an input file is processed by a RosettaNet connector, it is packaged and sent to the specified trading partner. When Arc receives a file over RosettaNet, it attempts to route the file to a specific RosettaNet connector. The application uses the RosettaNet identifiers in the RosettaNet message to determine which RosettaNet connector should receive the file. When a file is routed to a RosettaNet connector, that file is placed in the connector's **Transactions** tab, or is passed along to the next connector in the flow. ## Profile Configuration The RosettaNet Profile must be configured before connections can be established with individual RosettaNet connectors. Click **Profiles** on the navbar, then click the **RosettaNet** tab. ### RosettaNet Profile Tab #### Personal Id *Settings for identifying the local profile.* | Setting | Description | | - | - | | **Business Identifier** | Your identifier. Messages sent by Arc include this value in outgoing messages to identify the sender. Incoming messages must be addressed to this identifier. Identifiers are case-sensitive. | | **Business Location** | Describes the location of the receiver for incoming messages. This value is not an address, but it can be a city name. | #### Personal Certificate *Settings related to the private decryption and signature certificate.* | Setting | Description | | - | - | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Never share this certificate with external parties. Click the **Create Certificate** button to generate a self-signed certificate that is ready to use in a RosettaNet transaction: a corresponding public key is also generated with the same filename and a `.cer` extension. | | **Certificate Password** | The password required to access the **Private Certificate**. | #### Application URLs *Settings and displayed values related to accessing Arc from the public web.* | Setting | Description | | - | - | | **Receiving URL** | The URL at which Arc listens for incoming RosettaNet messages. Share this URL with all your trading partners. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Connector Configuration Once you configure the global RosettaNet profile settings, create and configure individual RosettaNet connectors for each trading partner on the [Flows](../flows/flows) page. ### Settings Tab #### Configuration *Settings for identifying and connecting to a specific RosettaNet trading partner.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Business Identifier** | The identifier specific to the target trading partner. This value is included in the headers for outgoing messages, and is also used to route incoming messages to the appropriate RosettaNet connector. | | **Business Location** | The location of the outgoing message recipient. | | **Partner URL** | The trading partner's public endpoint where outgoing messages should be sent. | #### Connection Info *Settings related to connection parameters for the specified trading partner.* | Setting | Description | | - | - | | **RNIF Version** | The protocol version to use when generating RosettaNet messages. | | **Encryption Type** | Whether to encrypt the payload, the payload and the headers, or neither. | | **Data Signature** | Whether to sign outgoing messages. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming messages. An error is thrown if a received message does not have a required security parameter. | | **Connection Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Test Indicator** | Controls the `GlobalUsageCode` in the preamble header. Check this to set the code to `Test`. If left unchecked, the code is `Production`. Only required when **RNIF Version** is 1.1. | #### Acknowledgments *Settings related to RosettaNet acknowledgments.* | Setting | Description | | - | - | | **Automatic Acknowledgments** | Whether acknowledgments should be automatically sent in response to incoming RosettaNet messages. | | **Automatic Exceptions** | Whether exceptions should be automatically sent in response to failed messages. | | **Require Acknowledgment** | Whether acknowledgments should be returned in response to outgoing RosettaNet messages. | | **Security** | Whether acknowledgments should include a signature block verifying the message integrity and identity of the recipient. | | **Delivery** | Whether acknowledgments should be returned as a direct response to the outgoing RosettaNet message (Synchronous) or returned later as part of a separate connection (Asynchronous). Synchronous receipts are recommended unless the size of the RosettaNet message is very large (50MB is a common threshold), in which case processing the message and delivering a synchronous receipt might strain the connection timeout duration. | #### Trading Partner Certificates *Settings related to the public key certificates provided by the trading partner.* | Setting | Description | | - | - | | **Encryption Certificate** | The public key certificate used for RosettaNet encryption when sending messages. This certificate must be paired with the trading partner's private decryption certificate, and the trading partner should provide this public key certificate when sharing RosettaNet configuration details. | | **Verification Certificate** | The public key certificate used to verify RosettaNet signatures when receiving messages. This field is usually unnecessary; most RosettaNet parties use the same private certificate for both signing and decrypting. If this field is not specified, the application uses the **Encryption Certificate** to verify signatures. | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the partner's RosettaNet system requires HTTPS instead of HTTP. If the trading partner does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or it can be set to `Any Certificate` to unconditionally trust the target server's identity. | ### Advanced Tab #### Alternate Local Profile *Settings that override the configuration on the **Profiles** page for this specific RosettaNet connector. Setting an alternate local profile lets you use different local certificates and identifiers for certain trading partners.* | Setting | Description | | - | - | | **Local Business Location** | Your location. Overrides **Business Location** on the Profiles page. | | **Local Business Identifier** | Your identifier. Overrides **Business Identifier** on the Profiles page. | | **Private Certificate** | The certificate used to decrypt incoming messages and sign outgoing messages. Overrides **Private Certificate** on the Profiles page. | | **Certificate Password** | The password required to access the local private certificate. | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Use Profile** | Whether to use the **Private Certificate** configured on the Profiles page as the TLS certificate for client authentication. | | **Private Certificate** | The private certificate presented during TLS client authentication. Only applicable if you are not using the private certificate on the Profiles page. | | **Certificate Password** | The password required to access the TLS client certificate. | #### HTTP Authentication *Settings related to HTTP client authentication.* | Setting | Description | | - | - | | **HTTP Authentication** | Whether to use client HTTP authentication. | | **HTTP Authentication Type** | Whether to provide HTTP authentication credentials in an encrypted format (Digest) or in plain text (Basic). Only use Basic authentication if the connection is an HTTPS connection (instead of HTTP). | | **User** | The user credential for HTTP client authentication. | | **Password** | The password credential for HTTP client authentication. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **RNIF Debug Mode** | Whether communication errors related to the validation of PIPs or the authentication of the sender should be returned by RosettaNet receivers. This setting must be enabled on both sides of the transfer for the exceptions to be returned. | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | | **Secure Transport** | Check this to require security when forwarding messages. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send Automation** | Whether files arriving at the connector are automatically sent as RosettaNet messages. | | **Retry Interval** | The number of minutes before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is based on a successful server acknowledgement and validation of the receipt (when requested synchronously). If you set this to 0, the connector retries the file indefinitely. | | **Resend Interval** | The number of minutes before unacknowledged messages are resent. A resend is triggered when the server receives the file, but an asynchronous acknowledgment is not provided within the expected timeframe. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when asynchronous acknowledgments are requested. Success is based on the return of an asynchronous receipt within the **Resend Interval** after a successful server acknowledgement. If a successful server acknowledgement is not returned, **Max Attempts** is applied instead. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Trading partners must provide some of the connection details that are required when you configure a new RosettaNet connector. At a minimum, these details should include: * Business Identifier * Partner URL * Partner Certificates ### Business Identifier Your trading partner is identified by their business identifier in a RosettaNet transaction. When sending outgoing requests, the business identifier is used in the header of the request to indicate the recipient. To establish a RosettaNet self-test, the identifier should be set to the same value as the **Business Identifier** on the [Profiles](../getting-started/administration/profiles) page. > [!NOTE] > > This value is case-sensitive. ### Partner URL The Partner URL is the endpoint where the trading partner receives RosettaNet transmissions. Outgoing RosettaNet messages are sent to this target endpoint, which must be unique for each trading partner. You can test the Partner URL with a web browser to check for networking or connectivity issues. To establish a RosettaNet self-test, the target URL should be identical or nearly identical to the **Receiving URL** on the Profiles page. You can replace the domain name from the Profiles page with the loopback address localhost to keep the RosettaNet transaction in the local network. An example local self-test URL is `http://localhost:8001/pub/Receive.rsb`. If you do not replace the domain name with localhost, the RosettaNet message is routed outside of the local network. You can use this to check network configuration settings and to make sure that the message can reach Arc through any firewalls. ### Partner Certificates Each RosettaNet connector must be configured with certificate(s) for the target trading partner. The trading partner provides the certificates necessary to encrypt and verify RosettaNet messages exchanged with them. Arc accepts X.509 public key certificates (files with .cer, .der, or .pem extensions). Typically the trading partner provides a single certificate, which should be configured in the **Encryption Certificate** field. If the trading partner provides multiple certificates, they should clarify the purpose of each certificate. If the partner provides a full certificate chain (as acquired from a commercial certificate authority), only the leaf certificate (the last certificate in the chain) needs to be configured. ## Send and Receive Files Once the RosettaNet profile and partner-specific RosettaNet connectors have been configured, files can be securely sent and received. ### Send Files In a RosettaNet connector, the **Transactions** tab displays the files to be sent to the target trading partner. If **Send Automation** is enabled on the [Automation](#automation-tab) tab, files that reach the **Transactions** tab of the connector are automatically packaged and sent. Access the log files for all transmissions by expanding the row associated with the transmitted file. On the **Transactions** tab, choose **More > Create Test Files** to generate a simple series of test files to send to the trading partner. ### Resend and Retry A RosettaNet Resend is triggered when the trading partner is expected to return an asynchronous acknowledgment, but fails to do so within the **Resend Interval** duration (60 minutes by default). The application then attempts to resend the transmission. The application continues resending the message until an acknowledgment is received or the **Max Attempts (async)** is exhausted. A Retry is triggered when the HTTP response from the trading partner indicates that the server has not received the transmission. This can indicate a networking or connectivity issue, which is often transient. The application retries the transmission every **Retry Interval** minutes until the transmission is received or the **Max Attempts** is exhausted. ### Receive Files In a RosettaNet connector, the **Transactions** tab displays the files that have been received by the application and routed to the connector (based on the RosettaNet identifiers present in the incoming RosettaNet message). Expand each file row to display a list of available logs for the transmission. These files are available on the connector **Transactions** tab. If the connector is connected to other connectors in the flow, files are automatically moved from the **Transactions** tab of the RosettaNet connector to the **Transactions** tab of the next connector in the flow. The RosettaNet protocol does not allow for actively pulling files from trading partners: the RosettaNet connector can only passively wait for a trading partner to send a file. ## Configuring PIPs Partner Interface Processes (PIPs) are the logical XML documents used by RosettaNet to transport information about various business processes. The following is a list of common PIPs: | PIP | Description | | - | - | | 3A1 | Request Quote | | 3A2 | Request Price and Availability | | 3A3 | Request Shopping Cart Transfer | | 3A4 | Request Purchase Order | | 3A5 | Query Order Status | | 3A6 | Distribute Order Status | | 3A7 | Notify of Purchase Order Update | | 3A8 | Request Purchase Order Change | | 3A9 | Request Purchase Order Cancellation | | 3A10 | Notify of Quote Acknowledgement | | 3A13 | Notify of Purchase Order Information | | 3A14 | Distribute Planned Order | | 3B1 | Distribute Transportation Projection | | 3B2 | Notify of Advance Shipment | | 3B3 | Distribute Shipment Status | | 3B4 | Query Shipment Status | | 3B5 | Request Shipment Change | | 3B6 | Notify of Shipments Tendered | | 3B11 | Notify of Shipping Order | | 3B12 | Request Shipping Order | | 3B13 | Notify of Shipping Order Confirmation | | 3B14 | Request Shipping Order Cancellation | | 3B18 | Notify of Shipment Documentation | | 3C1 | Return Product | | 3C2 | Request Financing Approval | | 3C3 | Notify of Invoice | | 3C4 | Notify of Invoice Reject | | 3C5 | Notify of Billing Statement | | 3C6 | Notify of Remittance Advice | | 3C7 | Notify of Self-Billing Invoice | ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# RSS Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/rss The RSS connector downloads remote RSS and Atom feeds with automatic polling, incremental update support, and multiple authentication schemes. RSS Connectors support downloading remote RSS and Atom feeds. ## Key Capabilities * RSS and Atom feed consumption with automatic polling and multiple authentication schemes * Incremental update support using If-Modified-Since headers and ETag-based duplicate detection * Configurable response header promotion for metadata extraction ## Overview RSS connectors expose a simple interface to automatically poll for downloadable content from a specific RSS or Atom feed. The connector can be configured to only process new and updated content, and can perform both HTTP and TLS authentication to the remote service. ## Connector Settings ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Feed URL** | The HTTP or HTTPS endpoint from where RSS content should be downloaded. | | **Feed Type** | Whether the remote URL is an RSS or Atom feed. | | **Use If-Modified-Since** | When enabled, the connector includes an If-Modified-Since header in the outgoing request that equals the latest publication date and time from the previous download attempt. This causes the server to filter responses based on what content is new or has been modified recently. | | **Process New Items** | When enabled, the connector stores the last modified date from previous downloads, and filters out results that are not new or modified since that date. The result is the same as **Use If-Modified-Since**, but the filtering is done on the client-side rather than sending a filter condition to the server. | #### Authentication *Settings related to authenticating with the RSS service.* | Setting | Description | | - | - | | **Authorization Type** | The type of authentication to use with the RSS service. See the [Authentication](#authentication) section for details. | #### TLS Server Authentication *Settings related to verifying the TLS server's identity.* | Setting | Description | | - | - | | **TLS Server Certificate** | The public key certificate used to verify the identity of a TLS server. You can upload a certificate, leave the field blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to trust the target server's identity. Use Any Certificate with caution: the certificate verifies that you are connecting to the intended server. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Include Channel** | Whether to include data about the RSS channel in the output. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2. | | **Skip Processed Items** | When enabled, the connector maintains a checksum of each item downloaded, and does not download the same content more than once. | | **Timeout** | The duration in seconds to wait for a response from the RSS server before throwing a timeout error. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Use ETag** | Servers might include a checksum of the RSS content as the ETag (entity tag). Enable this setting to have the connector include the ETag from the previous response as the If-None-Match header in the outgoing request. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Receive** | Whether the connector should automatically send RSS download requests according to a specified interval. | | **Interval** | The interval between automatic requests. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection A valid target **Feed URL** is required to establish a connection to any RSS service. Some services might also require authentication or a set of custom headers in order to consume the service. The connector silently supports both HTTP and HTTPS target URLs. If the target URL is an HTTPS URL, set the **TLS Server Certificate** to the public key certificate identifying the server. To implicitly trust the target endpoint, set the field to `Any Certificate`. ### Authentication The RSS connector supports HTTP authentication in the following formats: * Basic (plain text) * Digest (encrypted) * NTLM * Negotiate * Proprietary The connector provides the credentials specified in **Username** and **Password** with each request to the remote RSS service. ## Automatic Cache Cleanup Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. # S3 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/s3 Configuration and usage guide for the CData Arc S3 connector. The S3 connector integrates with Amazon's S3 (Simple Storage Service) and other S3-like services (such as Google Storage and Wasabi). ## Key Capabilities * Amazon S3 and S3-compatible service integration (Google Storage, Wasabi) with IAM role and access key authentication * Bucket-based file organization with bidirectional transfers and prefix-based virtual folders * Client-side and server-side encryption options with configurable access policies * Optional caching to ensure only new or updated files are downloaded ## Overview Each S3 connector can automatically upload to and download from a single S3 bucket. Before you begin, you need an Amazon account with the appropriate credentials (or account credentials for the S3-like service you are using). Specify the upload and download paths in the bucket. The connector supports download filters by file name. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to the remote connection target.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Service** | Use the dropdown to choose which service to connect to. Select **Other** to specify the base URL to use when connecting to the service. | | **Bucket Name** | The S3 bucket to poll or upload to. | | **Region** | The Region where the specified **Bucket Name** is stored. | #### Account Settings *Settings related to the account with permission to access the configured **Bucket Name**.* | Setting | Description | | - | - | | **IAM Role** | Whether to use the attached IAM role to access S3. Only use this setting when CData Arc is hosted on an EC2 instance that has an IAM role attached. The IAM credentials replace the two **Key** options below. | | **Access Key** | The Access Key account credential acquired from Amazon (or the S3-like service). | | **Secret Key** | The Secret Key account credential acquired from Amazon (or the S3-like service). | | **Assume Role ARN** | Use the two **Key** options above to call the Amazon STS service to obtain temporary credentials to access S3 with the provided role ARN. | #### TLS Settings *Settings related to TLS negotiation with the S3 server.* | Setting | Description | | - | - | | **TLS** | Check this to enable TLS negotiation. | | **Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the server requires a specific certificate for validation. If the server does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server's identity. | #### Upload *Settings related to the path in the specified bucket where files are uploaded.* | Setting | Description | | - | - | | **Prefix** | The remote path on the server where files are uploaded. | | **Overwrite Action** | Whether to overwrite, skip, or fail existing files. | #### Download *Settings related to the path in the specified bucket where files are downloaded.* | Setting | Description | | - | - | | **Prefix** | The remote path on the server from where files are downloaded. | | **File Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Access Policy** | The access policy set on objects after they are uploaded to the S3 server. | | **Encryption Password** | If set, object data is encrypted on the client side before upload, and downloaded objects are automatically decrypted. | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Server Side Encryption** | Whether to use server-side AES256 encryption. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Some obsolete operating systems do not support TLS 1.2. | | **Virtual Hosting** | Whether to use hosted-style or path-style requests when referencing the bucket endpoint. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The requirements for establishing an S3 connection are simple: * Amazon account credentials (or other S3-like account credentials) * **Access Key** * **Secret Key** * A bucket that can be accessed by the above account For Amazon S3, use [this link](https://aws-portal.amazon.com/gp/aws/securityCredentials) to obtain **Access Key** and **Secret Key** information from Amazon. Optionally, you can secure the connection with S3 servers with TLS by enabling the **Use TLS** option in the [TLS Settings](#tls-settings) section. ## Uploading ### Upload to Remote Folders The **Prefix** setting in the [Upload](#upload) section of the **Settings** page specifies the bucket path to upload files to. This allows for the logical separation of files into virtual folders in the same bucket. > [!NOTE] > > S3 servers do not maintain a real folder structure, and Arc uses application logic to present a pseudo folder structure. Slashes in the **Prefix** (`/`, `\\`) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from 'subfolders' in the bucket based on the slashes in the path. ### Upload Automation The S3 connector supports automatic upload via the [Automation tab](#automation-tab). When **Upload** automation is enabled, files that reach the **Transactions** tab for the connector are automatically uploaded to the specified **Bucket Name** at the specified **Prefix**. If a file fails to upload, the application attempts to send it again after the **Retry Interval** has elapsed. This process continues until the **Max Attempts** has been reached, after which the connector raises an error. ## Downloading ### Download from Remote Folders The **Prefix** setting in the [Download](#download) section of the **Settings** page specifies the bucket path to download files from. This allows for the logical separation of files into virtual folders in the same bucket. The **File Filter** setting provides a way to only download specific filenames in the specified path. > [!NOTE] > > S3 servers do not maintain a real folder structure, and Arc uses application logic to present a pseudo folder structure. Slashes in the **Prefix** (`/`, `\\`) are interpreted as representing a folder hierarchy. This allows for uploading to or downloading from 'subfolders' in the bucket based on the slashes in the path. ### Download Automation The S3 connector supports automatic download via the [Automation tab](#automation-tab). When **Download** automation is enabled, the connector automatically polls the remote bucket based on the specified **Download Interval**. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Schedule Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/schedule Add a time-based delay step to a flow, releasing messages to the next connector based on a configurable schedule interval. The Schedule connector adds a step to the flow that can be used to send messages to the next connector based on a schedule. ## Key Capabilities * Time-based flow control with configurable processing intervals (minutes, hours, or days) * Batch processing control with maximum file limits per interval * Scheduled message release for controlling downstream processing timing * Support for advanced cron expressions for complex scheduling requirements ## Overview Schedule connectors are configured with an interval. The automation service processes files on the connector **Transactions** tab after this interval has elapsed, instead of as soon as files are detected. This allows for control over the timing of file processing. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the scheduled delay of message processing.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Max Files** | By default, the connector processes all files on the **Transactions** tab at each send interval. Set this to a positive number to define the maximum number of files to process at each interval. | | **Max Files Per Hour** | By default, the connector does not limit the number of files processed in a one-hour window. Set this to a positive number to define the number of files to process across all scheduled sends in the one-hour period. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Interval** | The interval (such as Minutes, Hours, or Days) at which the connector processes files. For example, if the interval is set to 60 minutes, the connector moves all messages down the flow every 60 minutes, regardless of when those messages arrived at the connector. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # SCP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/scp Configuration and usage guide for the CData Arc SCP connector. The Secure Copy Protocol connector (SCP) transfers files to and from an SSH server via SCP. ## Key Capabilities * SCP file transfer over SSH with certificate-based and password-based authentication * Bidirectional file transfer with multiple remote path monitoring and glob pattern filtering * Configurable SSH encryption algorithms with automatic file deletion and retry logic ## Overview Each SCP connector uploads to a specified folder and downloads from a set of folders on the remote SSH server. Files that reach the SCP connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder(s) are downloaded and entered into the Arc flow. The connector supports upload and download filters to send and receive only specified file names and/or file extensions. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to establishing the connection to the remote SSH host.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Remote Host** | The hostname or IP address of the SSH server. | | **Remote Port** | The port on which to connect to the SSH server. | #### Client Authentication *Settings related to authenticating to the SSH server. Options vary based on the selected authentication mode.* | Setting | Description | | - | - | | **Authentication Mode** | The type of authentication to use with the SSH server. | | **User** | The username credential for logging in to the SSH server. | | **Password** | The password credential for logging in to the SSH server. | | **Client Certificate** | The certificate with a private key to use during public key authentication. | | **Certificate Password** | The password required to access the **Client Certificate**. | #### Server Authentication *Settings related to verifying the SSH server's identity.* | Setting | Description | | - | - | | **Server Thumbprint** | The thumbprint of the certificate that identifies the SSH server. You can also set this to a certificate file containing the server's public key. When you test the connection, the connector prompts to accept/trust the thumbprint presented by the server. | #### Upload *Settings related to uploading files to the SSH server.* | Setting | Description | | - | - | | **Remote Path** | Files processed by the connector are uploaded to this remote folder. | #### Download *Settings related to downloading from the SSH server.* | Setting | Description | | - | - | | **Remote Path** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | **File Mask** | A glob pattern that determines which files in the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete files (after download)** | Check this to delete successfully downloaded files from the SSH server. | ### Advanced Tab #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Delete Command** | The command to use when deleting files after download. | | **Max Download Files** | The maximum number of files to be downloaded from the SCP server in one polling interval. Set to **-1** to download all files. | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **SSH Encryption Algorithms** | A comma-delimited list of algorithms to use during SSH encryption. By default, all algorithms are enabled. The following algorithms are supported: aes256-ctr, aes256-cbc, aes192-ctr, aes192-cbc, aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, cast128-cbc, blowfish-cbc, arcfour, arcfour128, arcfour256, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Timeout** | The duration the connector will wait for a connection response before throwing a timeout error. | | **Upload File Permissions** | The file permissions to set on files uploaded by the connector. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Upload** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If this is set to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the remote download path(s) for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The following settings are required to connect to a remote SSH server: * Remote Host * Remote Port * Authentication Mode * User * Password or Client Certificate/Certificate Password * Server Thumbprint When you test the connection to the server, the connector prompts to accept/trust the thumbprint of the server's public key. ## Uploading Files The SCP connector uploads files from the **Transactions** tab. Files are automatically uploaded to the SSH server if **Upload Automation** is enabled in the connector. Set the **Remote Path** to the folder on the remote SSH server where files should be uploaded. The application changes directories to the specified folder after logging in to the server. If a file is not uploaded, the connector attempts to upload it again after the **Retry Interval** has elapsed. This process continues until the **Max Attempts** setting has been reached, after which the connector raises an error and the file is no longer sent automatically. ## Downloading Files Downloaded files are displayed in the **Transactions** tab of the connector, or passed along to the next connector in the flow. Files are automatically downloaded from the SSH server if **Download Automation** is enabled in the connector, and can be manually downloaded by clicking the **Receive** button in the **Transactions** tab. Set the **Remote Path** to the folder on the remote SSH server from which files should be downloaded. The application changes directories to the specified folder after logging in to the server. **File Mask** filters files so that only filenames matching the glob pattern are downloaded. You can use the **Delete files (after download)** setting to delete files on the remote server after they are successfully downloaded. On the **Transactions** tab, click **Receive** to test the settings and retrieve the matching files from the server. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Script Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/script Perform custom functions and operations using ArcScript or Python, including data transformations, external calls, and business rule processing. The Script connector provides the ability to perform functions or operations that are not otherwise available in the built-in connectors of CData Arc. ## Key Capabilities * Custom business logic implementation using [ArcScript](../scripting/arcscript) or [Python](../scripting/python-scripting) * Access to full message context including message headers and file content * Integration with external libraries and system commands (Python) * Support for complex data transformations and business rule processing * Batch group message processing using the [connectorBatchGroupRead](../scripting/operations/op-connector-batch-group-read) ArcScript operation ## Overview Script connectors use the powerful ArcScript scripting language to accomplish a wide range of custom behavior. The script configured in the connector is run each time a document is placed in the [Transactions](#transactions-tab) tab for the connector. The input document can be read and modified in the script, or the input document can simply trigger an external function configured in the connector. Script connectors can also be configured to run on a schedule via **Receive** automation on the [Automation](#automation-tab) tab. These are a few of the tasks you can configure a Script connector to do: * Replace values in the input document with data returned from custom SQL queries * Call a custom Web service to invoke a specialized workflow * Run an external script or executable via the sysExecute operation After you decide the appropriate task for your Script connector, read the Scripting documentation to explore how to call various operations using ArcScript. ## Connector Configuration This section contains all of the configurable connector properties. ### Script Tab *Options related to writing and testing the script.* ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/script_tab.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=86bc27cbd9e435a0606fd900363f7772) | Setting | Description | | - | - | | **Action dropdown** | Choose the connector's action. (See [Actions](#actions) for details on the three options.) | | **ArcScript editor** | Use the editor to write your script. The toolbar contains standard editing options such as undo, redo, delete, copy, and format. It also includes an option to hide the Insert Options in the right pane. | | **Insert Options** | The panel in the right pane makes it easy to insert code into your script with one click. The dropdown includes **Inputs and Outputs** (such as ConnectorId or Encoding; the available options depend on the configured Action), **Snippets**, Keywords, Operators and Formatters. Find the code you want to insert, and click to add it. | | **Testing tab** | Specify whether to view the script input and/or output as the message body or headers. Options on this tab vary based on the script's action type (for example, trigger scripts do not have inputs). Click **Test** to test the script. | | **Log tab** | Displays the test log. | | **Last update on** | Displays the datetime the script was last modified and saved. | | **Auto update** | Use the toggle to automatically test the script after each modification. | #### Actions This connector can perform any of the three connector action types: • *Trigger* executes a script on a schedule and might produce files to be sent down the flow. This action serves as the starting point of a flow. • *Transform* accepts messages from the flow as input for a script and produces output. This action serves as the middle of a flow. • *Terminal* accepts messages from the flow as input for a script and acts as the terminal point of the flow. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | | **Retry Interval** | The number of minutes that the connector waits before retrying a failed script execution. | | **Max Attempts** | The number of times that the connector retries a failed script execution before raising an error. | | **Receive** | Whether the connector should automatically run the script on a configured schedule. | | **Receive Execution Interval** | The unit of measure to use for the receive automation schedule (such as Minutes, Hours, or Days). | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. Options on this tab vary depending on the connector's [action type](#actions): * If your connector is a **Trigger**, use the **Receive** button to start the flow. * If your connector is a **Transform** or **Terminal**, use the **Upload Files** button to upload files to the flow. ## Using the arc:push Keyword The [arc:push](../scripting/keyword-reference/op-arc-push) keyword is commonly used in Script connectors to pass items to the output feed of the script. In addition, when you select an **Output** attribute from the Insert Options menu, if an `arc:push` keyword is not already included in the script, Arc auto-injects it so the script can work. Following are three common use cases. 1 Create and output a file: ```xml theme={null} ``` 2 Pass an input file as output: ```xml theme={null} ``` 3 Write entries to the Script connector log file: ```xml theme={null} ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# SFTP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sftp Configuration and usage guide for the CData Arc SFTP connector. The SFTP connector transfers files to and from a remote SFTP server. ## Key Capabilities * SSH-based secure file transfer with public key and password authentication * Advanced authentication including certificate-based client authentication * Resume capability for interrupted uploads and downloads * Advanced SSH encryption algorithms and server thumbprint validation * Connection pooling and temporary file handling for reliable transfers * Optional caching to ensure only new or updated files are downloaded ## Overview Each SFTP connector uploads to a specified folder and downloads from a set of folders on the remote SFTP server. Files that reach the SFTP connector in the Arc flow are uploaded to the upload target folder, and files found in the download target folder(s) are downloaded and entered into the Arc flow. The connector supports upload and download filters to only send and receive specified file names and/or file extensions. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to establishing the connection to the remote SFTP host.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Remote Host** | The hostname or IP address of the SFTP server. | | **Remote Port** | The port on which to connect to the SFTP server. | #### Client Authentication *Settings related to authenticating to the SFTP server.* | Setting | Description | | - | - | | **Authentication Mode** | The type of authentication to use with the SFTP server. | | **User** | The username for logging in to the SFTP server. | | **Password** | The password for logging in to the SFTP server. | | **Client Certificate** | The certificate with a private key to use during public key authentication. Only available when **Authentication Mode** is set to **Public Key** or **Multi-Factor Authentication**. Optionally, you can use the **Create Certificate** button to create a new certificate. | | **Client Certificate Password** | The password required to access the **Client Certificate**. | #### Server Authentication *Setting related to verifying the SFTP server's identity.* | Setting | Description | | - | - | | **Server Thumbprint** | The thumbprint of the certificate that identifies the SFTP server. You can also set this to a certificate file containing the server's public key. When you test the connection, the connector prompts you to accept/trust the thumbprint presented by the server. | #### Upload *Setting related to uploading files to the SFTP server.* | Setting | Description | | - | - | | **Remote Path** | Files processed by the connector are uploaded to this remote folder. You can use macros to dynamically create dated folders (such as year or month folders) in your path. For more information, see [Macros](#macros). | #### Download *Settings related to downloading from the SFTP server.* | Setting | Description | | - | - | | **Remote Path** | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. You can use macros to dynamically create dated folders (such as year or month folders) in your path. For more information, see [Macros](#macros). | | **File Mask** | A glob pattern that determines which files in the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete files (after received)** | Check this to delete successfully downloaded files from the SFTP server. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Server Commands These options allow you to execute custom commands on the target server at various times during connector execution. > [!NOTE] > > While the commands described below are commonly accepted, be sure to check with the target server administrator to ensure that the server can accept a command before trying to use it.
SettingDescription
**After Connect**Execute this command directly after connecting.
**After Get**Execute this command after downloading each file.
**After Put**Execute this command after uploading each file.
**Before Get**Execute this command before downloading each file.
**Before Put**Execute this command before uploading each file.
Following are the commands you can submit to the target server when you include them in the fields described above: • `onerror`: Pass the `fail` argument to instruct the connector to treat errors that come from the custom scripts as errors for the overall transaction. Without `fail` the connector logs any exceptions and continues processing. • `cd`: Change directory. • `mkdir`: Create directory. • `rn` or `move`: Rename or move file. • `del` or `rm`: Delete or remove file. • `rmdir`: Remove directory. CData Arc supports the following macros with these commands. They all use the following syntax: `%Macro%`. Some macros, such as `%Date%` and `%SourceFilename%`, do not require an argument, but others do. The macros that take an argument use the following syntax: `%Macro:argument%`.
MacroDescription
MsgHeader:*headername*Evaluates to the header value of the message (for example: %MsgHeader:headername%).
SourceFilenameEvaluates to the filename (extension included) of the file currently being processed.
SourceFilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed.
DateEvaluates to the date the file was last modified in a yyyy-MM-dd format (for example, 2024-02-13).
DateTimeEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss format (for example, 2024-02-13T08:14:22).
TimeEvaluates to the time the file was last modified in a HH:mm:ss format (for example, 08:14:22).
DateTime.tzEvaluates to the datetime the file was last modified in a yyyy-MM-dd'T'HH:mm:ss zzz format (for example, 2024-02-13T08:14:22 PST).
Time.tzEvaluates to the time the file was last modified in a HH:mm:ss zzz format (for example, 08:14:22 PST).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
**Example**: After downloading, you want to move each file from the download folder to a History subfolder. Use the **After Get** field to supply the `move` command with the `%SourceFilename%` macro. ![Server command example](/26.4/images/server_command_example.png) #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Connection Pooling** | Check this to have the connector utilize a pool to access and store connections. This is useful in scenarios where the server places restrictions on the frequency of logon attempts. | | **Download Excluded Extensions** | A comma-delimited list of file extensions that should not be downloaded by the connector. | | **Download Excluded Prefixes** | A comma-delimited list of prefixes that instructs the connector not to download filenames that begin with a specified prefix. | | **Max Download Files** | The maximum number of files to be downloaded from the SFTP server in one polling interval. Set to **-1** to allow all files to be downloaded. | | **Minimum File Size** | The minimum file size to process, in bytes. If a file being uploaded is smaller than the specified size, it is finalized as **Skipped**. For files being downloaded, the connector adds a transaction log entry indicating that the file was skipped because of its size. | | **Maximum File Size** | The maximum file size to process, in bytes. If a file being uploaded is larger than the specified size, it is finalized as **Skipped**. For files being downloaded, the connector adds a transaction log entry indicating that the file was skipped because of its size. | | **Monitor File Growth** | Check this to have the application store the file sizes of the remote files between polling intervals, and only download files that do not change in size. This prevents the connector from downloading files that are still in the process of being written on the server. | | **Recurse** | Check this to download files in subfolders of the target remote path. The subdirectories are preserved for the received files. | | **Resume Downloads** | If enabled, when an error occurs during a file download, the connector stores the portion of the file that was downloaded and attempts to download the remainder of the file on the next download attempt. | | **Resume Uploads** | If enabled, when an error occurs during a file upload, the connector notes the number of bytes that were transferred and attempts to upload the remainder of the file on the next upload attempt. | | **SSH Encryption Algorithms** | A comma-delimited list of algorithms to use during SSH encryption. By default, all algorithms are enabled. Arc supports the following algorithms: aes256-ctr, aes256-cbc, aes192-ctr, aes192-cbc, aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, cast128-cbc, blowfish-cbc, arcfour, arcfour128, arcfour256, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com) | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Receive Filter** | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, \*.txt). You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | **Temp Download Extension** | A temporary file extension to append to file downloads while the transfer is in progress. | | **Temp Upload Extension** | A temporary file extension to append to file uploads while the transfer is in progress. | | **Temp Upload Prefix** | A temporary file prefix to prepend to file uploads while the transfer is in progress. | | **Temp Upload Path** | A temporary path on the remote SFTP server where files are uploaded. The file is moved into the permanent upload path after the transfer is complete. | | **Timeout** | The length of time (in seconds) the connector waits for a response from the server before throwing a timeout error (the default is 60 seconds). | | **Wait for Channel Close** | Check this to have the connector wait for the server to close the SSH channel before disconnecting. If timeouts occur during the logoff sequence, you should disable this setting. | | **Use GMT File Time** | Check this to have the connector read file timestamps as GMT time, and to use that time for comparisons against the local cache. If you leave this unchecked, timestamps from the server are converted to local time before being used for time comparisons. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Upload** | Whether files arriving at the connector should automatically be uploaded. | | **Retry Interval** | The number of minutes before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Download** | Whether the connector should automatically poll the remote download path(s) for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The following settings are required to connect to a remote SFTP server: * Remote Host * Remote Port * Authentication Mode * User * Password or Client Certificate/Certificate Password * Server Thumbprint When testing the connection to the server, the connector prompts you to accept/trust the thumbprint of the server's public key. ## Uploading Files The SFTP connector uploads files from the **Transactions** tab. Files are automatically uploaded to the SFTP server if **Upload Automation** is enabled in the connector. **Remote Path** should be set to the folder on the remote SFTP server where files should be uploaded. You can use [macros](#macros) to dynamically create dated folders (such as year or month folders). The application changes directories to the specified folder after logging in to the server. If any files could not be uploaded, the connector attempts to send them again after **Retry Interval** minutes have elapsed. This process continues until the **Max Attempts** setting is reached, after which the connector raises an error and the file is no longer sent automatically. ## Downloading Files Downloaded files are displayed in the **Transactions** tab of the connector, or passed along to the next connector in the flow. Files are automatically downloaded from the SFTP server if **Download Automation** is enabled in the connector, and can be manually downloaded by clicking the **Receive** button on the Transactions tab. **Remote Path** should be set to the folder on the remote SFTP server from which files should be downloaded. You can use [macros](#macros) to dynamically create dated folders (such as year or month folders). The application changes directories to the specified folder after logging in to the server. The **File Mask** property filters files so that only filenames matching the glob pattern are downloaded from the server. You can use the **Delete files (after download)** setting to delete the files on the remote server after successfully downloading them. On the Transactions tab, click the **Receive** button to test the settings and retrieve the matching files from the server. ## Subfolder Headers If **Recurse** is checked on the **Advanced** tab, when receiving a file from the **Remote Path** in the SFTP connector: * The message for the file that is received contains a Subfolder header in its metadata. * This header contains the subfolder, relative to the **Remote Path** in the SFTP connector, that the file was received from. * This subfolder header is supported by other connectors that support a subfolder header in Send operations. When sending files, if a Subfolder header is present on the message that is sent to the **Upload Path** in the SFTP connector: * The file is placed in the subfolder, relative to the **Upload Path** in the SFTP connector, that is specified in the subfolder header. * The subfolder is created if permissions are available. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# SFTP Server Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sftp-server Configuration and usage guide for the CData Arc SFTP Server connector, which defines client profiles for authenticating to the built-in SFTP server. Each SFTP Server connector defines a unique client profile that can be used to authenticate to the Arc SFTP Server. ## Key Capabilities * Embedded SSH-based secure file server with multi-user support * Flexible authentication including password, public key, and Windows/AD integration (Windows/AD integration is only available in the .NET edition) * Individual user directories with customizable folder structures and additional paths * Advanced security features including login attempt lockouts and IP restrictions * Support for temporary file extensions and automatic file movement after operations ## Overview The Arc SFTP Server connector is primarily configured on the [Profiles](../getting-started/administration/profiles) page. Then, individual SFTP Server connectors can be created for each trading partner who should have access to the server. The SFTP Server connector defines a trading partner's credentials (username, password and/or public key) and provides a unique home directory on the server. Each user's home directory contains a *Send* folder, where clients can download files, and a *Receive* folder, where clients can upload files. You can rename these folders on the [Advanced tab](#advanced-tab). SFTP clients are not given permissions to the root of the SFTP server, which means that SFTP clients should always *cd* into the Send or Receive directories after connecting. The SFTP Server also supports Windows/AD authentication. See [Windows Authentication](#windows-authentication) for details. ### Video Resources Watch this short video for an overview of how to configure an SFTP Server. [https://www.youtube.com/embed/R1OrVVdqySU](https://www.youtube.com/embed/R1OrVVdqySU) ## Profile Configuration The SFTP Server profile must be configured before connections can be established with individual SFTP Server connectors. Click **Profiles** on the navbar, then click the **SFTP Server** tab. ### Server Configuration *Server implementation settings.* | Setting | Description | | - | - | | **Port** | The port on which the SFTP server listens for incoming connections. | | **Server Certificate** | The certificate that identifies the server. | | **Certificate Password** | The password required to access the server certificate. | | **Login Banner** | The banner presented to SFTP clients when they connect to the server. | | **Root Directory** | The root directory for the server. Subfolders are created in the root for individual client profiles (for each configured SFTP Server connector). Each client profile includes a Send folder, where clients can download files from the server, and a Receive folder, where clients can upload files to the server. | | **Windows Authentication** | Check this to use Windows Authentication to authenticate users instead of the SFTP Server connector configurations. Only available in the [.NET edition](../getting-started/installation/windows-edition) of Arc. See [Windows Authentication](#windows-authentication) for more information. | | **Security Group** | The name of the Windows group used for granting access to the server. This can be a group on the local machine or on the domain. Only applicable when **Use Windows Authentication** is checked. | #### Lockouts *Optional settings related to locking server access.* | Setting | Description | | - | - | | **Failed Attempts** | The number of unsuccessful login attempts allowed before the user is locked out. | | **Lockout Period** | The length of time (in minutes) that the user is locked out. | | **Time Check Period** | The length of time (in minutes) that records are kept of failed login attempts. | #### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature:
  • `localhost` cannot be modified or removed from the list.
  • Any IP addresses outside of the defined ranges are rejected.
  • Ranges are supported. For example, the entry `100.10.100.1-15` indicates that IP addresses between `100.10.100.1` and `100.10.100.15` are allowed.
  • Classless inter-domain routing (CIDR) notation is supported. For example, the entry `100.10.100.0/24` indicates that IP addresses between `100.10.100.0` and `100.10.100.255` are allowed.
  • Wildcard patterns are supported. For example, the entry `100.10.100.*` indicates that IP addresses beginning with `100.10.100` are allowed.
> [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Rotate Interval** The number of days to wait before creating a new log file.
**Log Delete Interval** The number of days to wait before deleting old log files.
#### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Inactivity Timeout** | The length of time (in seconds) that must pass without activity before a user times out. | | **Allowed Files Filter** | A glob pattern that determines which files are accepted by the SFTP server. You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Separate multiple file types by commas (for example, \*.x12,\*.edi). | | **Encryption Algorithms** | When negotiating algorithms, each side sends a list of all the algorithms it supports or allows. The algorithm chosen for each direction is the *first* algorithm to appear in the sender's list that the receiver supports, so provide multiple algorithms as a comma-separated list in preferential order. The available algorithms are: `aes256-ctr`, `aes192-ctr`, `aes128-ctr`, `aes256-cbc`, `aes192-cbc`, `aes128-cbc`, `3des-ctr`, `3des-cbc`, `blowfish-cbc`, `arcfour256`, `arcfour128`, `arcfour`, `cast128-cbc`, `aes256-gcm@openssh.com`, `aes128-gcm@openssh.com`, and `chacha20-poly1305@openssh.com`. The field is empty by default, which means that all encryption algorithms except `arcfour`, `arcfour128`, and `arcfour256` are enabled. | | **Other Settings** | Enables you to configure hidden profile settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal use cases and functionality should not require the use of these settings. | ## Connector Configuration Once you configure the SFTP Server profile settings, create and configure an individual SFTP Server connector for each trading partner on the Flows page. ### Settings Tab #### Configuration
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Server Status *The current state of the connector's underlying service. The following statuses are available:*
Status Description
**Active** The service is running.
**Inactive** The service is configured but is not currently running.
**Not Configured** The underlying profile or service has not been set up.
Click the **View SFTP Server Profile** link to open the profile in a new tab. #### User Configuration *Credentials for authenticating to the local SFTP server.* | Setting | Description | | - | - | | **User** | The username credential for logging in to the local SFTP server. | | **Authentication Mode** | The type of authentication to use with the SFTP server. The following fields vary based on your authentication mode. | | **Password** | The password credential for logging in to the SFTP server. | | **Client Certificate** | The public key certificate corresponding to the private certificate the client uses when you choose **Public Key** authentication. | #### Permissions *Settings related to the read/write permissions for the Send and Receive folders.* | Setting | Description | | - | - | | **Send Directory Permissions** | Use the checkboxes to set read/write permissions for the *Send* directory. This is where files are downloaded. | | **Receive Directory Permissions** | Use the checkboxes to set read/write permissions for the *Receive* directory. This is where files are uploaded. | ### Advanced Tab #### Local Folders *Settings related to the folders where clients upload and download files. Rename the default folders here.* | Setting | Description | | - | - | | **Input Folder (Send)** | Files placed in the Send folder are available to be downloaded by clients. | | **Output Folder (Receive)** | Files uploaded by the client should be placed in the Receive folder. Files remain in the Receive folder or are passed along to the next connector in the flow. | #### Additional Paths The SFTP Server connector allows you to expose paths in addition to the Input and Output folders. To configure additional paths, follow these steps: 1. Use the **Path** field to specify the additional path that should be exposed. Path values are relative to the **Root Directory** defined on the Profiles page. 2. Set the permissions for the additional path using the **Read** and **Write** checkboxes. 3. If more paths are needed, click **New** and repeat these steps for each path. For example, if **Root Directory** is set to `/var/opt/arc/sftpserver`, and an additional path of `MyAdditionalPath` is added, it maps to the `/var/opt/arc/sftpserver/MyAdditionalPath` path on disk. #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Home Directory** | The configured server path where connected users start after logging in. Use the connector home, the Send or Receive folder, or a defined [additional path](#additional-paths). For the valid locations and how the path is validated, see [Home Directory](#home-directory). | | **Allowed Files Filter** | A glob pattern that determines which files are accepted by the SFTP server. You can use negative patterns to indicate files that should *not* be downloaded (for example, -\*.tmp). Separate multiple file types by commas (for example, \*.x12,\*.edi). Overrides the **Allowed Files Filter** option on the Server Configuration portion of the Profiles page. | | **Move File After Send** | Specifies whether files in the *Send* folder should be moved to the *Sent* folder after they are downloaded by the client. | | **Temp Receive Extensions** | Files with a matching extension are not recorded in the Receive table and do not fire the **After Receive** event until after they are renamed. Supply a comma-delimited list of extensions. | | **Timeout** | The length of time (in seconds) the server waits for a connection response before throwing a timeout error. | | **Save Subfolder** | Check this to have a *Subfolder* header added to received messages. It represents the path relative to the local folders or additional paths. | | **Finalize on Disconnect** | Check this to delay file finalization until the client disconnects. This allows clients to perform additional operations (such as renaming) on uploaded files without errors caused by the file being moved prematurely. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection Each SFTP Server connector represents a single trading partner's connection parameters. The trading partner should connect to the SFTP server using the server settings from the [Profiles](../getting-started/administration/profiles) page (port, server certificate, and so on) and the authentication settings in the dedicated SFTP Server connector (user and password). Each trading partner has a pair of *Send* and *Receive* directories that are subfolders of the root. The partner downloads files from the Send folder and uploads files to the Receive folder. The client is not permitted to upload or download files from the root. ## Home Directory The **Home Directory** setting controls which folder a client lands in when it logs in to an SFTP Server connector. Configure it on the connector's **Advanced** tab, under **Advanced Settings**. Setting a home directory is useful when you migrate from other SFTP or FTP software that lets you configure a landing folder, or for automated systems that upload or download at whatever directory they land in without issuing navigation (`cd`) commands. The field defaults to empty, but it also accepts a server path: * **Empty (default):** The client lands at the connector home (`/`). * **Set to a path:** On successful login, the client's initial working directory is the specified path. A client that connects and immediately uploads, downloads, or lists files without changing directories operates inside the home directory. The home directory must be a server path (starting with `/`) that resolves against the **Root Directory** defined on the connector's [Profile](#profile-configuration), and it must match a configured server location; you cannot set it to an arbitrary path that only exists on disk. For the SFTP Server connector, the valid locations are: * `/` (the connector home): this is the default when the field is empty * `//Send` * `//Receive` * Any defined [additional path](#additional-paths) (for example, `/Staging`) The server root (`/`) is not a valid home directory on the SFTP Server connector; it returns a permission error, because there is no readable mount at the root. The field normalizes your input, so `Receive`, `/Receive`, `\Receive`, and `/Receive/` are all treated as the same path. The value is validated when you save the connector: a path that does not match a configured server location is rejected with an error that names the invalid value. If you set an additional path as the home directory and that path is later removed from the configuration, saving the connector rejects the now-stale value. > [!NOTE] > > The home directory only sets the initial landing directory at login. It is not a chroot or jail and does not restrict which folders a client can navigate to afterward; folder access is still governed by the [permission settings](#permissions). ### Home Directory and the SFTP Client Remote Path When an [SFTP](./sftp) client connector connects to an SFTP Server connector that has a home directory configured, the SFTP client's **Remote Path** setting still applies independently. The following table shows the resulting working directory for each combination: | SFTP Server Home Directory | SFTP Client Remote Path | Resulting working directory | | - | - | - | | *(empty)* | *(empty)* | `/` | | *(empty)* | `/` | Permission error | | *(empty)* | `Receive` | `//Receive` | | `/Test` | *(empty)* | `/Test` | | `/Test` | `/` | Permission error | | `/Test` | `/` | `/` | ## Windows Authentication When Windows Authentication is enabled on the [Server Configuration](./sftp-server#server-configuration) portion of the Profiles tab, individual SFTP Server connectors are not required to grant login access to the SFTP Server. Instead, you need to specify the name of the Windows **Security Group** that should be granted access to the server. When Windows Authentication is enabled, the **Root Directory** profile setting supports the %User% and %Domain% macros to establish separate root directories for separate users in the security group. Therefore, when Windows Authentication is enabled, users *are* permitted to upload/download files in the root directory (this is not true when you use SFTP Server connectors for authentication). Once files are uploaded to the user-specific folder, they can be entered into the Arc flow using a [File connector](./file). ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## Common Errors ### Error: Could not bind server socket: Permission denied. **Cause** This error can appear when attempting to connect to an SFTP server and the process hosting Arc does not have sufficient privileges to establish a listener on the specified port. Note that in some cases (such as Linux environments and hosted instances running in an Amazon AMI), ports below 1024 are forbidden from access. **Resolution** Choose a different port, or change the identity of the process hosting Arc to one with permissions to bind to the port. The Amazon AMI-hosted version of Arc uses the Ubuntu operating system, so CData recommends that you use an *Uncomplicated Firewall* (UFW) to manage port permission issues. For example, setting up SFTP Server to run on port 8022 in Arc and using UFW to forward port 22 to 8022 at the OS level looks like this: ``` ufw allow 22/tcp ufw allow 8022/tcp echo " *nat :PREROUTING ACCEPT [0:0] -A PREROUTING -p tcp --dport 22 -j REDIRECT --to-port 8022 COMMIT" >> /etc/ufw/before.rules ``` If your environment uses a different Linux operating system, CData recommends that you bind to a port above the restricted range (for example, 8022 for SFTP traffic) and use `iptables` to route incoming requests on the desired port to the allowed port: `iptables -t nat -I PREROUTING -p tcp --dport 22 -j REDIRECT --to-port 8022` # SharePoint Connection Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sharepoint/sharepoint Connect CData Arc to Microsoft SharePoint for document management and file transfer. The SharePoint connector allows you to integrate SharePoint into your data flow by pushing or pulling data from SharePoint. ## Key Capabilities * Microsoft SharePoint document management integration with multiple authentication schemes (OAuth, ADFS, Azure AD, and NTLM) * Document library access for file upload and download with overwrite and deletion control * Optional caching to ensure only new or updated files are downloaded ## Prerequisites You must first establish a connection to SharePoint through one of two methods: * Add a SharePoint connector to your flow. Then, on the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the CData Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **SharePoint**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to SharePoint can be reused across multiple SharePoint connectors. ## Enter Connection Settings 1. Enter the connection information: | Field | Description | | - | - | | Name | The static name of the connection. Set this as desired. | | Type | This is always set to SharePoint. | | Auth Scheme | The authentication scheme to use for the connection. Options are ADFS, AzureAD, AzureServicePrincipalCert, AzurePassword, AzureMSI, Basic, Negotiate, None, NTLM, OAuth, OAuthJWT, Okta, OneLogin, PingFederate, and SharePointOAuth. | | URL | The URL to the SharePoint site you are connecting to. | 2. Enter the connection information for your chosen authentication scheme: * [OAuth](#oauth) * [All other auth schemes](#other-auth-schemes) 3. Select your newly-created connection in the **Connection** drop-down list of the connector configuration pane. ### OAuth 1. If needed, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 2. Click **Connect**. 3. On the new tab that opens, log into your SharePoint account. If prompted, provide access permissions. 4. Return to Arc. * If the connection succeeds, click **Add Connection**. * If the connection fails, check your URL and login credentials and try again. ### Other Auth Schemes 1. In the **User** field, enter the username to use for logging in. 2. In the **Password** field, enter the password for the user. 3. If needed, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 4. Click **Test Connection** to ensure that Arc can connect to SharePoint with the provided information. 5. Click **Add Connection** to finalize the connection. ## Send and Receive Configuration After you create a connection to SharePoint, see [SharePoint Configuration](./sharepoint-config) for instructions on configuring the connector for sending and receiving files. # SharePoint Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sharepoint/sharepoint-config Configure the SharePoint connector for sending and receiving files in CData Arc. Once you have created a connection to SharePoint, the following sections help you configure the connector so you can send and receive files. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration See [SharePoint Connection Setup](./sharepoint) for detailed instructions on how to create a connection to SharePoint. #### Send Configuration Settings related to uploading to the remote storage. | Setting | Description | | - | - | | Remote Folder | Files processed by the connector are uploaded to this folder. Hover over the field to display a list of supported macros. | | Overwrite | Whether to overwrite or skip existing files. | #### Receive Configuration Settings related to downloading from the remote storage. | Setting | Description | | - | - | | Remote Folder | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. | | File Mask | A glob pattern that determines which files in the Remote Path should be downloaded (for example, `*.txt`). | | Delete | Check this to delete successfully downloaded files from the remote storage. | #### Caching Settings related to caching and comparing files between multiple downloads. | Setting | Description | | - | - | | File Size Comparison | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | Timestamp Comparison | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. > > By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. > > When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### Advanced Settings Settings not included in the previous categories. | Setting | Description | | - | - | | Recurse Remote Subdirectories | Whether to download files in subfolders of the target remote path. | | Chunked Upload Configuration | Whether to upload large files in chunks using SharePoint upload sessions (required for files larger than 250MB). | | Chunk Size | The chunk size (in MB) for uploading large files. The minimum is 1MB and the default is 10MB. | | Receive Filter | A glob pattern filter to determine which files should be downloaded. You can use negative patterns to indicate files that should not be downloaded (for example, `-*.tmp`). Use this setting when you need multiple **File Mask** patterns. Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. | | Local File Scheme | Instructs the connector to group files in the local folder according to the selected interval. | | Processing Delay | The amount of time (in seconds) the connector waits before processing a file. | #### Miscellaneous | Setting | Description | | - | - | | Timeout | The length of time (in seconds) the server waits for a connection response before throwing a timeout error. | #### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | #### Logging | Setting | Description | | - | - | | Log Level | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | Log Subfolder Scheme | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option instructs the connector to create a new subfolder each week. | | Log Messages | Check this to have the log entry for a processed file include a copy of the file itself. | #### Other Settings | Setting | Description | | - | - | | Other Settings | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ### Automation Tab #### Automation Settings Settings related to the automatic processing of files by the connector. | Setting | Description | | - | - | | Send | Whether files arriving at the connector are automatically uploaded. | | Retry Interval | The amount of time before a failed upload is retried. | | Max Attempts | The maximum number of times the connector processes the input file. If set to 0, retries indefinitely. | | Receive | Whether the connector should automatically poll the remote download path for files to download. | | Interval | The interval between automatic download attempts. | | Minutes Past the Hour | The minutes offset for an hourly schedule. Only applicable when the interval is Hourly. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | Time | The time of day for the attempt. Only applicable when Daily, Weekly, or Monthly. | | Day | The day for the attempt. Only applicable when Weekly or Monthly. | | Minutes | The number of minutes to wait. Only applicable when Minute interval. | | Cron Expression | A five-position cron expression. Only applicable when Advanced interval. | #### Performance | Setting | Description | | - | - | | Max Workers | The maximum number of worker threads consumed from the threadpool. | | Max Files | The maximum number of files sent by each thread. | ### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Sending and Receiving ### Send Files to SharePoint To send files, set the **Remote Folder** in the [Send Configuration](#send-configuration) portion of the **Settings** page to the folder where files should be uploaded. Each SharePoint connector uploads to a single folder location. Use the **Overwrite remote files** option to indicate whether files that already exist in the container should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the remote storage. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. ### Receive Files from SharePoint To receive files, set the **Remote Folder** in the [Receive Configuration](#receive-configuration) portion of the **Settings** page to the folder from where files should be downloaded. Use a comma-delimited list to specify multiple folders. Use the **File Mask** to set a glob filter (for example `*.txt`) that limits the downloads to files matching the filter. To specify multiple filters, use the **Receive Filter** option on the [Advanced tab](#advanced-tab). Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. ## Macros | Macro | Description | | - | - | | ConnectorID | The Connector ID of the connector. | | Ext | The file extension of the file (e.g., `xml`, `csv`). | | Filename | The original filename of the message. | | FilenameNoExt | The original filename of the message, without the file extension. | | MessageId | The unique identifier assigned to the message. | | RegexFilename | A filename derived from a regex on the original filename. | | Header | The value from a message header, referenced by name. | | LongDate | The current date/time in long format. | | ShortDate | The current date/time in short format. | | DateFormat | The current date/time in a custom format. | | Vault | The value from a vault entry, referenced by name. | # SOAP Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/soap Send SOAP messages to remote web services and retrieve responses, with WSDL import, WS-Security, and multiple authentication methods. SOAP connectors support sending SOAP messages to remote web services and retrieving the response. ## Key Capabilities * Full SOAP web service client with WSDL import for automatic configuration * WS-Security support with message signing, encryption, and certificate management * Multiple authentication methods including HTTP, Token, and TLS client authentication * SOAP template generation through WSDL integration with the [XML Map](./xml-map/xml-map) connector * Support for SOAP versions 1.1 and 1.2 with configurable security token formats ## Overview The input to a SOAP connector should be the full SOAP message to send to the web service. You can generate a template for the SOAP message by importing a WSDL for the target web service. Read the [SOAP Templates](#soap-templates) section for details. The SOAP connector supports [HTTP Authentication](#http-authentication), [Token Authentication](#token-authentication), and [TLS Client Authentication](#tls-client-authentication), and can add digital encryption and signatures to the outgoing SOAP message for web service security. After sending the SOAP request, the connector stores the response or passes it along to the next connector in the CData Arc flow. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Partner Info *Settings for identifying the target web service.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Service Endpoint** | The URL to which SOAP requests should be sent. | | **Allow ArcScript in Service Endpoint** | Allows for the evaluation of ArcScript expressions in the service endpoint URL before the query is issued. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](https://mintcdn.com/cdata-arc/fuF73A8feEsIWbbA/26.4/images/rest_arcscript_editor.png?fit=max\&auto=format\&n=fuF73A8feEsIWbbA\&q=85\&s=05d3576b0141d612fcc6e83469592e28)), which enables you to build dynamic URLs using ArcScript. | | **SOAP Action** | The web service action to be triggered on the remote server. | | **SOAP Version** | The version of SOAP you are using. | | **Import from WSDL** | WSDLs can be imported from a URL or from a local file, and then automatically populate configuration fields in the connector. Importing a WSDL also allows for generating an XML template for SOAP requests when connected to an XML Map connector. See [SOAP Templates](#soap-templates) for details. | #### WS-Security *Settings related to securing the web service transmissions.* | Setting | Description | | - | - | | **Send Message Security** | Whether to sign and/or encrypt outgoing SOAP requests. | | **Receive Message Security** | Whether to require that signatures and encryption are present for incoming SOAP responses. An error is thrown if a received message does not have a required security parameter. | | **Time To Live** | The length of time the connector waits for a connection response before throwing a timeout error. | #### Client Certificate *Settings related to the private local certificates.* | Setting | Description | | - | - | | **Private Certificate** | The private certificate used for signing outgoing requests and decrypting incoming responses. | | **Certificate Password** | The password for accessing the private certificate. | #### Server Certificates *Settings related to the server's public certificates.* | Setting | Description | | - | - | | **Encryption Certificate** | The public certificate used for encryption when sending messages and signature verification when receiving responses. | | **TLS Server Certificate** | The public certificate used to verify the identity of an SSL/TLS server. Only required if you are sending requests over HTTPS (instead of HTTP). In the absence of a specific server certificate, you can leave this blank to allow the underlying OS/JVM to perform certificate validation, or you can set it to `Any Certificate` to trust the target server's identity. Use Any Certificate with caution: the certificate verifies that you are connecting to the intended server. | ### Advanced Tab #### Token Authentication *Settings related to token client authentication.* | Setting | Description | | - | - | | **Token Authentication** | Check this to enable token authentication for outgoing messages. If enabled, outgoing messages include the user credentials set below. | | **Username** | The username required to access the token. | | **Password** | The password required to access the token. | | **Password Type** | Use the radio buttons to specify whether the password is in plain text (**Text**) or is encrypted (**Digest**). | #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.*
SettingDescription
**Private Certificate** The private certificate presented during TLS client authentication.
**Certificate Password** The password required to access the TLS client certificate.
#### HTTP Authentication *Settings related to HTTP client authentication.* | Setting | Description | | - | - | | **Use HTTP Authentication** | Check this to enable client HTTP Authentication. | | **HTTP Authentication Type** | Whether to provide HTTP authentication credentials in an encrypted format (Digest) or in plain text (Basic). Basic authentication should only be used if the connection is an HTTPS connection (rather than HTTP). | | **User** | The User credential for HTTP client authentication. | | **Password** | The Password credential for HTTP client authentication. | #### Custom Headers A set of custom headers to include as part of the outgoing message. #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Connection Timeout** | The length of time the connector waits for a connection response before throwing a timeout error. | | **Encryption Algorithm** | The algorithm to use when encrypting outgoing messages. | | **Normalize Issuer Subject** | Whether certificate issuer data should be normalized into a simple string for validation by the web service. | | **Signature Algorithm** | The algorithm to use when signing outgoing messages. The same algorithm is requested for the corresponding MDN receipts. | | **Sign Header Payload** | By default, the connector only signs the body of the SOAP message. Check this to have the connector also sign the headers. | | **Security Token Format** | Determines the format used for security tokens in SOAP payloads. **X509** references contain the certificate subject in DN (distinguished name) format. **Binary** tokens use the Base64 encoded contents of the entire public key. **KeyIdentifier** references contain the X509 Subject-Key-Identifier format. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **TLS Enabled Protocols** | The list of SSL/TLS protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. Keep in mind that SSLv2 and SSLv3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support them. | | **Verification Certificate** | The public certificate used to verify incoming SOAP responses. This field is only necessary if a different certificate should be used to verify signatures than the public certificate set in **Encryption Certificate**. | | **Output Behavior** | By default, the connector outputs a message containing the `Response` data. The other options allow you to output the `Input Message` so it can be processed further, or to not output a message at all. See [Response Event](#response-event) for more information. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as SOAP requests. | | **Retry Interval** | The amount of time before a failed send is retried. A retry is triggered when the server does not respond to a send attempt, or responds negatively to communicate that the file was not received. | | **Max Attempts** | The maximum number of times the connector sends outgoing requests. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the request indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The **Service Endpoint** and **Service Action** for the target web service are required to establish a connection. Some web services might require one or more security features for SOAP requests: * Encryption/Signing (WS-Security) * Client SSL Authentication * HTTP Authentication ## SOAP Templates The SOAP connector can simplify the process of generating SOAP requests through the use of a WSDL file and an [XML Map](./xml-map/xml-map) connector that is connected to the SOAP connector in the flow. The WSDL file defines the appropriate XML structure for SOAP requests, and the XML Map connector can transform any XML structure into this target structure. After using the **Import WSDL** button to import the file, connect an XML Map connector to the SOAP connector (so that files processed by the XML Map connector are then sent to the SOAP connector). After saving these flow changes, the XML Map connector can dynamically read the WSDL XML and use it as a **Destination File** template. Then, upload any source XML structure as the **Source File** in the XML Map connector. After the **Source File** and **Destination File** are set in the XML Map connector, the visual designer is populated with the source and destination structures. Drag-and-drop elements from the source to the destination to map the relationship between the two XML structures. After the mapping is complete, files that match the **Source File** are automatically converted into SOAP requests by the XML Map connector. ## Response Event You can use a response event to interact with the response received from the server (which includes the body, headers, cookies, and so on), and to enrich the output message generated from the connector. You can use the following special items in the `Response` event. * [\_cookies](../scripting/introduction-to-arcscript#http-cookies-_cookies) * [\_message](../scripting/introduction-to-arcscript#message-_message) * [\_request](../scripting/introduction-to-arcscript#http-request-_request) * [\_response](../scripting/introduction-to-arcscript#http-response-_response) ### Response Event Example This script reads the XML response received from the server, parses out the access token contained in the XML, and adds it as a header onto the output message created by the connector: ```xml theme={null} ``` The following steps detail what happens: 1. The body of the response sent by the server back to the connector is accessed via `_response.body` and is set to the text attribute of the xmlDOMGet operation. Other attributes of xmlDOMGet are also populated, such as the `map` attribute with the XPath to the desired token in the response body. 2. The xmlDOMGet operation is called. If the token is found in the response XML body, it is added as a message header to the output message of the connector via the `_message.header:access_token` syntax. If the token is not found, the value of the `access_token` header is set to a static string: *Token not found!*. This type of script is useful when you need to parse data from the raw XML response body received back from a server after a request is sent to it. The header can then be read and used in subsequent connectors in the flow. > [!TIP] > > If the server responds using JSON, you can achieve the same result using jsonDOMGet. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Split Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/split Split a single XML file into multiple XML files using XPath-based rules, with support for batch grouping and wildcard element matching. The Split connector splits a single XML file into multiple XML files. ## Key Capabilities * XPath-based XML file splitting for batch record separation into individual files * Configurable batch grouping with wildcard XPath support for dynamic element matching * Max records per file control for output file size management ## Overview The Split connector is configured with an XPath upon which input XML files should be split into multiple output XML files. This is useful when an XML file contains a batch of data such as multiple orders, multiple line items, or multiple customer records. The Split connector splits this batched XML data into a separate XML file for each order, item, or record. See the [Split Examples](#split-examples) for more information. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **XPath** | The path in the XML structure to the element upon which the split should occur. Each unique occurrence of the specified XPath results in a unique output XML file. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Size** | Controls how many messages are put into a batch group. The default is 1, which means a batch group is not created. If you set it to a value less than 1, all messages are contained in a single batch group. If you set it to a value greater than 1, each batch group contains the number of messages indicated here. See the [Split Examples](#split-examples) for more information. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Split Examples Below is an example XML input file that contains multiple *TransactionSet* elements (in other words it contains a batch of transactions): ```xml theme={null} 1 value1 value2 value3 value4 ``` The Split connector can split this input file into four separate output files, one for each *TransactionSet* element. To accomplish this, set the **XPath** field to the XPath to the *TransactionSet* element: `/Items/Interchange/TransactionSet` An output file for each *TransactionSet* would result. One example is shown below. Output 1: ```xml theme={null} 1 value1 ``` If you set **Batch Size** to 2, the connector outputs two batch groups, each with two messages. One example batch group is shown below. Batch 1 Batch Message 1: ```xml theme={null} 1 value1 ``` Batch Message 2: ```xml theme={null} 1 value2 ``` ## XPath Wildcards The **XPath** can include a wildcard character (\*) to split on all elements at a given XPath. For example, the input XML might contain multiple groups of data that need to be split into separate files, but the groups of data have different element names: ```xml theme={null} value1 value2 value3 ``` Split these groups by setting the **XPath** to the following value: `/Items/*` The following three output files would result: Output 1: ```xml theme={null} value1 ``` Output 2: ```xml theme={null} value2 ``` Output 3: ```xml theme={null} value3 ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# TCP Server Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/tcp-server Host a generic TCP/IP endpoint for receiving arbitrary data streams, with configurable message delimitation, TLS/SSL support, and bidirectional communication. TCP Server connectors support hosting a generic endpoint for receiving data over TCP. ## Key Capabilities * Generic TCP/IP server endpoint with TLS/SSL and plain text support for arbitrary data streams * Configurable message delimitation using static delimiters or length-based parsing * Bidirectional communication with Connection ID-based routing and connection event messaging ## Overview Each TCP Server connector listens on a specified port for incoming TCP traffic, either with TLS/SSL encryption or plain text. Use this connector when CData Arc needs to receive arbitrary data that does not necessarily fit into the constraints provided by other standard protocols. Since TCP traffic is a constant stream that does not intrinsically have a start or end point, you must configure the connector to determine where one message ends and the next begins. The connector supports two approaches: * A static string/character delimiter that ends the current message or begins the next message * Reading a value in the incoming TCP data to determine how long the message should be For the second approach, the connector requires an offset (how many bytes after the first byte to begin reading the message length) and a size (how many bytes to read, starting at the offset, to get the length value). If the length of the message is the first value in the header for the message, the offset is 0. Once the delimiter or length is reached, the connector pushes a message into the Arc flow containing the raw data received over TCP. The TCP Server connector can also generate an Arc message when a client connects or disconnects, to alert other processes within the flow. See the [Connected and Disconnected Messages](#connected-and-disconnected-messages) section for more information. For information on sending responses back to TCP clients, see the [Sending Messages](#sending-messages) section. ## Connector Settings ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Port** | The port on which to listen for incoming TCP connections. | | **Server Status** | The state of the connector's server: **Active** (the server is running; click **Stop Server** to stop it), **Inactive** (the server is not running; click **Start Server** to start it). To review the logs the server produces while it runs, see the [Logs tab](#logs-tab). | #### Message Decoding *Settings related to distinguishing individual messages from the TCP stream.* | Setting | Description | | - | - | | **Message Start Delimiter** | The single character or hexadecimal value (prefixed by `0x`) indicating the start of a new message. | | **Message End Delimiter** | The single character or hexadecimal value (prefixed by `0x`) indicating the end of the current message. | | **Message Length Offset** | The number of bytes to skip before beginning to read the length of the message from the incoming data. | | **Message Length Size** | The size in bytes of the length value in the incoming data. | #### TLS Settings *Settings related to TLS/SSL transport security.* | Setting | Description | | - | - | | **Enable TLS** | Whether clients must negotiate TLS/SSL encryption to connect to the TCP server. | | **Server Certificate** | The TLS/SSL certificate that identifies the TCP server. | | **Certificate Password** | The password to access the server certificate's private key. | ### Advanced Tab #### Keep Alive *Settings related to keep-alive packets sent by the server to maintain an idle connection.* | Setting | Description | | - | - | | **Keep Alive** | Whether to send keep-alive packets periodically to ensure that idle connections are not closed due to inactivity. | | **Keep Alive Time** | The length of inactivity that should elapse before sending the first keep-alive packet. | | **Keep Alive Interval** | The interval between consecutive keep-alive packets. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Connected Messages** | Whether the connector should generate an Arc message when a client connects. See the [Connected and Disconnected Messages](#connected-and-disconnected-messages) section for more information. | | **Disconnected Messages** | Whether the connector should generate an Arc message when a client disconnects. See the [Connected and Disconnected Messages](#connected-and-disconnected-messages) section for more information. | | **Idle Timeout** | The length of time (in seconds) the server should wait for an idle client before disconnecting them due to inactivity. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Local Host** | The binding address of the local server, if not the default network interface. | | **Max Connections** | The maximum number of concurrent connectors. | | **TLS Enabled Protocols** | The list of TLS/SSL protocols supported when establishing outgoing connections. Best practice is to only use TLS protocols. SSL v2 and SSL v3 are considered vulnerable and should only be used if your partner does not support higher versions. Keep in mind that TLS v1.3 is not universally adopted, and might be refused if the destination server does not support it. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Server Tab #### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature:
  • `localhost` cannot be modified or removed from the list.
  • Any IP addresses outside of the defined ranges are rejected.
  • Ranges are supported. For example, the entry `100.10.100.1-15` indicates that IP addresses between `100.10.100.1` and `100.10.100.15` are allowed.
  • Classless inter-domain routing (CIDR) notation is supported. For example, the entry `100.10.100.0/24` indicates that IP addresses between `100.10.100.0` and `100.10.100.255` are allowed.
  • Wildcard patterns are supported. For example, the entry `100.10.100.*` indicates that IP addresses beginning with `100.10.100` are allowed.
> [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. ### Automation Tab #### Automation *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Logs Tab The **Logs** tab lists the service logs generated by this connector. Each row represents a log file and shows its date modified, file name, size, and status. Use the search bar or the status filter to find a specific log, click a file name to preview or download it, or click **Download All Logs** to download all of them as a single zip file. ## Connected and Disconnected Messages By default, the TCP Server connector only generates Arc messages when clients transfer data that the connector recognizes as a full message (according to the settings in the [Message Decoding](#message-decoding) section). The connector can also generate Arc messages when a client connects to the server (if **Connected Messages** is enabled on the [Advanced tab](#advanced-tab)) or disconnects from the server (if **Disconnected Messages** is enabled on the Advanced tab). When these special messages are generated, a special header is added to the Arc message: `x-tcpserver-event` This header is set to the value `connected` when clients are connecting, and `disconnected` when clients are disconnecting. This header value must be checked by later connectors in the flow to detect whether a message written to the output folder of a TCP Server connector is one of these special messages. ## Sending Messages The TCP Server connector also uses a special header value to send messages back to connected clients: `x-tcpserver-connectionid` This header is added to any output messages generated by the connector, and is set to a Connection ID value that uniquely identifies the connected client. Messages that should be sent back to this client *must* have this header set to the same value to ensure that the message is sent to the appropriate client. Since this header is already present on any messages written out by the connector, if the message passes through an Arc flow and arrives back at the **Transactions** for the TCP Server connector, it still has the same Connection ID value and is returned to the client that sent the original message. If the message cannot be directly routed back to the TCP Server connector, the Connection ID value must be saved and applied as a header (with the name shown above) to any new messages that should be returned to the client. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# TRADACOMS Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/tradacoms Configuration and usage guide for the CData Arc TRADACOMS connector. TRADACOMS connectors support generating TRADACOMS documents from XML and converting TRADACOMS documents into XML. ## Key Capabilities * UK EDI standard support with bidirectional TRADACOMS/XML translation and automatic schema detection * Transmission header management with configurable delimiters and encoding options * Batch transaction support for multi-document interchange processing ## Overview When receiving TRADACOMS documents, TRADACOMS connectors validate interchange headers and convert the TRADACOMS document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The TRADACOMS connector automatically reads the input file to determine the appropriate schema, then parses the document according to this schema. When generating TRADACOMS documents, TRADACOMS connectors convert XML into TRADACOMS document syntax and apply the appropriate interchange headers. This is useful as the final step for creating a TRADACOMS document, after the XML data has been fetched and transformed elsewhere in the flow. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert TRADACOMS documents into XML, or XML data into TRADACOMS documents. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner ID (FROM Code)** | Identifies the party sending the TRADACOMS document. When generating TRADACOMS from XML, this should be your code. Only applicable when **Dynamically process partners** is unchecked. | | **Personal Code (UNTO Code)** | Identifies the party receiving the TRADACOMS document. When converting TRADACOMS into XML, this should be your code. Only applicable when **Dynamically process partners** is unchecked. | #### Transmission Settings *Settings related to the transmission headers of TRADACOMS documents. When generating documents, these settings are applied as interchange headers in the resulting document. When parsing documents, the interchange settings are used to validate the incoming document.* | Setting | Description | | - | - | | **Syntax Identifier (STDS Identifier)** | Identifies the syntax rules used in the TRADACOMS document. | | **Partner Name (FROM Name)** | The name of the party sending the TRADACOMS document (optional). When generating TRADACOMS from XML, this should be your name. | | **Personal Name (UNTO Name)** | The name of the party receiving the TRADACOMS document (optional). When converting TRADACOMS into XML, this should be your name. | | **Partner Reference (SNRF)** | The Id used to reference the sender of the TRADACOMS document. | | **Personal Reference (RCRF)** | The Id used to reference the recipient of the TRADACOMS document. | | **Application Reference (APRF)** | The Id used to reference the TRADACOMS application for this document exchange. | | **Transmission Priority Code (PRCD)** | The code representing the priority of the TRADACOMS transmission. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Release Char** | The character that *releases* (escapes) the next character, overriding its usual meaning. This allows reserved characters to appear as data within documents, as long as they are preceded by the Release Char. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An interchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the interchange. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is TRADACOM to XML. | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing an EDI qualifier include child elements containing the qualifier code and value. Only applicable when the **Translation Type** is TRADACOM to XML. For example: ``   `ST`   `Ship To` `` | | **Generate Description As** | When translating TRADACOM into XML, descriptions of the TRADACOM segments and elements can be provided as context for the TRADACOM data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects EDI structures that have hierarchical relationships embedded in the EDI data, and generates XML with these hierarchical relationships represented as parent-child relationships. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Track Transaction Types** | Whether to add transaction types as tracked headers to processed messages. | | **Validate Identifiers** | Check this to ensure that the identifiers in the translated document match the identifiers in the connector's configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_tradacoms_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_tradacoms_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## TRADACOMS to XML When generating XML, set the **Translation Type** to *TRADACOMS to XML*. TRADACOMS files placed in the **Transactions** tab are processed by the connector. The connector validates TRADACOMS message headers against the configured [Transmission Settings](#transmission-settings). After validating the transmission, the TRADACOMS segments are converted into XML according to the schema specified in the message. The resulting XML files are placed in the connector **Transactions** tab. ## XML to TRADACOMS When generating TRADACOMS files, set the **Translation Type** to *XML to TRADACOMS*. XML files placed in the **Transactions** tab are processed by the connector. The connector converts the XML elements representing a TRADACOMS file into TRADACOMS segments, then uses the configured [Transmission Settings](#transmission-settings) to generate the transmission headers. To create an example of the XML structure required to generate TRADACOMS documents, start with an example TRADACOMS document and send it through the TRADACOMS connector in *TRADACOMS to XML* mode. You can use the resulting XML as a template in the [XML Map connector](./xml-map/xml-map) to convert XML structures into TRADACOMS structure. ## Use With the XML Map Connector In accordance with Arc convention, the TRADACOMS connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound TRADACOMS documents, or as the following step in the flow when receiving inbound TRADACOMS documents. The interaction between the XML Map connector and the TRADACOMS connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the TRADACOMS connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the TRADACOMS connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the TRADACOMS document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# TSV Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/tsv Convert Tab-Separated Values (TSV) files to XML and generate TSV files from XML, with configurable delimiters, encoding, and column mapping. The TSV connector converts Tab-Separated Values (TSV) files into XML and generates TSV files from XML. ## Key Capabilities * Bidirectional TSV/XML conversion with configurable delimiters and encoding * Header detection and custom column mapping capabilities * Support for various character encodings and special character handling ## Overview XML is the primary format that CData Arc uses to manipulate data within a flow. Therefore, it is useful to convert TSV files into XML as a staging step for further processing in the flow, or to convert XML to a TSV file after the XML has been manipulated. Both of these operations can be accomplished with the TSV connector. For details on converting TSV into XML, see the [Converting TSV to XML](#converting-tsv-to-xml) section. For details on converting XML into TSV, see the [Converting XML to TSV](#converting-xml-to-tsv) section. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the connector Id and description.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Connector Settings *Settings related to the core operation of the connector.*
SettingDescription
**Column headers present** Whether the TSV file contains a row of headers providing names or context to the values in the file.
**Record Name** The name of elements representing a row in the TSV file when converting to XML. You can use the following macros: %ConnectorID%, %FilenameNoExt%, %RegexFilename:%, and %Header:%. See [Converting TSV to XML](#converting-tsv-to-xml) for more details.
#### Other Settings *Settings not included in the previous categories.*
SettingDescription
**Local File Scheme** A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros).
**Line Separator** The character sequence to use to denote line endings. By default, this is determined by the system, but you can use this setting to override the system behavior. Use `CR` or `\r` to denote carriage return, and `LF` or `\n` to denote line feed.
### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**.
  1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**.
  2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**.
  3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**.
### Advanced Tab
SettingDescription
**Processing Delay** The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.*
SettingDescription
**Send** Whether messages arriving at the connector are automatically processed.
#### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting TSV to XML When a TSV file is transformed into XML, the resulting XML has the following structure: ```xml theme={null} ``` Each row (record) in the original file becomes a child of the root element `Items`. The name of all record elements is determined by the **Record Name** option set in [Connector Settings](#connector-settings). Each record element then has child elements corresponding to the values in each row of the input file. Some TSV files include a line of header information that provides context to the values in the file. When the **First line is header information** setting is enabled, this header line is parsed, and the parsed headers are used as the element names for the value elements (the children of the record elements). Otherwise, the value elements are given generic names such as `field_0`, `field_1`, and so on. ## Converting XML to TSV To convert XML to a TSV file, the input XML must have a 'flat' structure. This means that, disregarding the `Items` root element, the depth of the XML structure is two. For example: ```xml theme={null} Citizen Kane 1941 119 Sharknado 2013 86 ``` The XML is interpreted as follows: * Children of the root element are treated as records (rows) in the resulting file * Children of each record element are treated as the values in each row If the **First line is header information** option is enabled, a header row is inserted into the resulting TSV file with the names of each value element to provide context to the values. In the example above, this header row would consist of `title`, `year`, and `runtime`. ## Print View When you view a TSV document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![](/26.4/images/tsv_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## TSV Transformation: Using the XML Map Connector Many data transformation flows use the TSV connector in conjunction with the [XML Map Connector](/26.4/self-hosted/en/connectors/xml-map/xml-map). Often, data enters an Arc flow in TSV format and should exit the flow in some other format (for example, a database insert, an EDI file, or an insert into a CRM or ERP data source), or vice versa. Arc uses a single streamlined approach to these data transformation requirements:
  • Model the input format as XML
  • Model the output format as XML
  • Use the XML Map connector to map between the input XML and the output XML
Therefore, the TSV connector is commonly adjacent to an XML Map connector in the flow:
  • When TSV files are the *input* to the flow, the TSV connector converts a TSV file to XML and then passes that XML off to the XML Map connector to be transformed
  • When TSV files are the *output* from the flow, the TSV connector receives XML from the XML Map connector and converts it into a TSV file
An XML Map connector requires a sample XML structure for both the mapping input/source and the mapping output/destination. Use the [Sample Files](#sample-files-tab) tab to upload a sample file. Next, connect a TSV connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the TSV connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > The structure of this test file should be representative of future files. In other words, all of the TSV files processed by the TSV connector (and then the XML Map connector) should have the same columns as your test file. You might need to set up multiple TSV connectors and multiple XML Map connectors to handle distinct TSV structures. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Validate Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/validate Perform XML or message header validation using configurable XPath-based rules to catch invalid data before it causes errors later in a flow. Validate connectors perform an XML or message header validation step in order to catch invalid XML or message headers before they cause an error later in the flow. ## Key Capabilities * XML and message header validation using configurable rules and conditions * XPath-based validation for XML content with sample file support for rule creation * Data type validation for strings, numbers, and DateTime values * Group-based rule organization with complex logical conditions (AND/OR) ## Overview Validate connectors are configured with one or more rules that determine whether input XML or message headers are considered valid. Valid input successfully passes through the connector, while XML files or message headers that fail validation throw an error and are not passed along in the flow. For each configured rule, the connector checks the value from the input XML or message header against the target value to see if it is considered valid. See [Rules Tab](#rules-tab) for details. ## Connector Configuration This section contains all of the configurable connector properties. ### Rules Tab *Add and configure rules and groups of rules. Input XML and message headers are considered valid if they conform to all of the configured rules.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Sample File** | (optional) A file that represents the structure of the XML being validated. If you upload a sample file, the rules editor displays a tree representation of the document so you can easily find the XPath you need. | You can build complex validations by defining individual rules and/or groups of rules. To add a validation rule, click **Add Rule** or **Add Group**. When you build your rules, use the dropdowns to make your selections and choose your operator. You can choose from the following: * Input XML * Value: The value at the XPath being evaluated * Data Type: The data type of the value (String, Number, DateTime) * Occurrences: The number of times the XPath occurs in the document * Message Header * Value: The value of the message header being evaluated * Data Type: The data type of the value (String, Number, DateTime) When you select an XPath for a rule, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/validate_select.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=be8acee3bcffcb55fcc08c0f1342f522) You can reorder rules and groups. Click the handle ![handle](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_condition_handle.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=29b06331be6308858fbb8a3d38a42f2c) and drag the object to a new location. Input XML is only considered valid if all validation rules (or groups of rules) are satisfied. If the XPath configured in a rule is not found in the input XML, the validation automatically fails. In other words, if the connector is unable to read a value for a rule's XPath, the input file does not pass the rule. ### Settings Tab #### Connector Settings
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # VDA Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/vda Configuration and usage guide for the CData Arc VDA connector. VDA connectors support generating VDA documents from XML and converting VDA documents into XML. ## Key Capabilities * German automotive EDI standard support with bidirectional VDA/XML translation * Automatic schema detection and validation with interchange header generation * Support for VDA-specific segment structures, nested loops, and qualifier expansion ## Overview When receiving VDA documents, VDA connectors validate interchange headers and convert the VDA document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The VDA connector automatically reads the input file to determine the appropriate schema, then parses the document according to this schema. When generating VDA documents, VDA connectors convert XML into VDA document syntax and apply the appropriate interchange headers. This is useful as the final step for creating a VDA document, after the XML data has been fetched and transformed elsewhere in the flow. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector should convert VDA documents into XML, or generate VDA documents from XML. | #### Header Settings *Settings related to the VDA interchange headers. When converting documents into XML, the document headers are validated against these settings; when generating documents from XML, these settings are used to generate document headers.* | Setting | Description | | - | - | | **Sender Identifier** | The unique Id identifying the sending party in the VDA communication (when generating a VDA document, this should be your identifier). | | **Receiver Identifier** | The unique Id identifying the receiving party in the VDA communication (when generating a VDA document, this should be your partner's identifier). | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Suffix** | Appears at the end of segments to distinguish them from each other. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An exchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the exchange. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is VDA to XML. | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing a VDA qualifier include child elements containing the qualifier code and value. For example: ``   `ST`   `Ship To` `` | | **Functional Acks** | By default, all functional acknowledgments (997, 999) are routed to the connector selected in the flow diagram, and XML translations are not received in the **Transactions** tab. Check this to have the translated acknowledgment also be included in the **Transactions** tab. This allows functional acknowledgments to be integrated into a destination source in addition to new EDI documents. | | **Generate Description As** | When translating VDA into XML, descriptions of the VDA segments and elements can be provided as context for the VDA data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects VDA structures that have hierarchical relationships embedded in the VDA data, and generates XML with these hierarchical relationships represented as parent-child relationships. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## VDA to XML When generating XML, set the **Translation Type** to *VDA to XML*. The connector then processes VDA files placed in the **Transactions** tab. The connector validates VDA message headers against the **Sender Identifier** and **Receiver Identifier** configured in the [Header Settings](#header-settings). After validating the transmission, the VDA segments are converted into XML according to the schema specified in the message. The resulting XML files are placed in the connector's **Transactions** tab. ## XML to VDA When generating VDA files, set the **Translation Type** to *XML to VDA*. The connector then processes XML files placed in the **Transactions** tab. The connector converts the XML elements representing a VDA file into VDA segments, then uses the **Sender Identifier** and **Receiver Identifier** fields configured in the [Header Settings](#header-settings) to generate the transmission headers. To create an example of the XML structure required to generate VDA documents, start with an example VDA document and send it through the VDA connector in *VDA to XML* mode. You can use the resulting XML as a template in the [XML Map connector](./xml-map/xml-map) to convert XML structures into a VDA structure. ## Use With the XML Map Connector In accordance with Arc convention, the VDA connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound VDA documents, or as the following step in the flow when receiving inbound VDA documents. The interaction between the XML Map connector and the VDA connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the VDA connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the VDA connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the VDA document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# WebDAV Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/webdav Configuration and usage guide for the CData Arc WebDAV connector. WebDAV connectors support uploading resources to and downloading resources from an HTTP server. ## Key Capabilities * HTTP-based file transfer with multiple authentication schemes (None, Basic, Digest, NTLM, and Negotiate) * Bidirectional file synchronization with remote folder polling and file mask filtering * TLS client authentication and proxy support for secure connections * Optional caching to ensure only new or updated files are downloaded ## Overview Each WebDAV connector is configured with an upload path and download paths on the remote server. Files processed by the connector are automatically uploaded to the specified upload path, and the download paths are polled for resources to retrieve. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Authorization *Settings related to authenticating with the remote connection target.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **URL** | The URL of the remote HTTP server host. | | **Auth Scheme** | The authentication scheme to use for the connection. Options are None, Basic (the default option), Digest, NTLM, Negotiate, and Proprietary. | | **User Name** | The username credential for authenticating with the server. | | **Password** | The password credential for authenticating with the server. | | **Authorization Token** | The token required for authorization when **Auth Scheme** is set to **Proprietary**. | #### Server Authentication *Settings related to authenticating the identity of a TLS/SSL server.* | Setting | Description | | - | - | | **Server Public Certificate** | The public key certificate used to verify the identity of a TLS/SSL server. This is only necessary if the server requires a specific certificate for validation. If the server does not provide a TLS server certificate, you can leave this setting blank to allow the underlying OS/JVM to perform certificate validation, or set it to `Any Certificate` to unconditionally trust the target server's identity. | #### Send *Settings related to uploading files to the remote server.* | Setting | Description | | - | - | | **Remote Folder** | Files processed by the connector are uploaded to this folder. | | **Overwrite Action** | Whether to overwrite or skip existing files. | #### Receive *Settings related to downloading resources from the remote server.* | Setting | Description | | - | - | | **Remote Folder** | The folder on the remote server that is polled for files to download. Use a comma-delimited list to specify multiple paths. | | **File Mask** | A glob pattern that determines which files within the **Remote Path** should be downloaded (for example, \*.txt). | | **Delete** | Check this to delete successfully downloaded files from the remote storage. | #### Caching *Settings related to caching and comparing files between multiple downloads.* | Setting | Description | | - | - | | **File Size Comparison** | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. | | **Timestamp Comparison** | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. | > [!NOTE] > > Arc automatically cleans up the resource cache. This process runs in the background during every receive cycle and removes stale cache entries for files that are no longer present on the remote server. This keeps the cache size manageable without requiring any manual action. By default, files are retained for 30 days, but you can use the `ResourceCacheRetentionDays` setting in the **Other Settings** field on the **Advanced** tab to adjust it to fit your environment. For example, `ResourceCacheRetentionDays=15` retains files for 15 days before removing them. When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between `TEST.TXT` and `test.txt`. ### Advanced Tab #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.* | Setting | Description | | - | - | | **Client Certificate** | The private certificate presented during TLS client authentication. | | **Client Certificate Password** | The password required to access the TLS client certificate. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Excluded Extensions** | A comma-delimited list of file extensions that should not be processed by the connector. | | **Temp Upload Extension** | A temporary extension to append to file uploads. After the upload completes, the file is renamed to its file name. | | **Recurse** | Whether to download files in subfolders of the target remote path. | | **Timeout** | The length of time (in seconds) the server waits for a connection response before throwing a timeout error. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically uploaded. | | **Retry Interval** | The amount of time before a failed upload is retried. | | **Max Attempts** | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically poll the HTTP server for files to download. | | **Interval** | The interval between automatic download attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection To establish a connection, the WebDAV connector requires the **URL** of the target HTTP server and any authentication credentials required by the server. Set the **Auth Scheme** property to your authentication mechanism, and the **Settings** tab automatically updates with the appropriate set of credentials to enter. ## Uploading and Downloading ### Upload To upload files, set the **Remote Folder** in the [Send](#send) portion of the **Settings** page to the path on the remote HTTP server where files should be uploaded. Use the **Overwrite remote files** option to indicate whether files that already exist on the server should be overwritten or skipped. After configuration is complete, files placed in the connector **Transactions** tab are uploaded to the HTTP server. If **Send Automation** is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab. If a file fails to upload, the application attempts to send it again after the **Retry Interval** has elapsed. This process continues until the **Max Attempts** has been reached, after which the connector raises an error. ### Download To download files, set the **Remote Folder** in the [Receive](#receive) portion of the **Settings** page to the path on the remote HTTP server from where files should be downloaded. Use a comma-delimited list to specify multiple folders. Use the **File Mask** to set a glob filter (for example \*.txt) that limits the downloads to files matching the filter. Use **Delete** to specify whether successfully downloaded files should be removed from the remote storage. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Webhook Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/webhook Expose a public API endpoint for receiving HTTP POST and PUT data, with HMAC authentication, rate limiting, CORS support, and custom response scripting. Webhook connectors support exposing a public API endpoint. ## Key Capabilities * Public API endpoint exposure for HTTP POST and PUT data ingestion with modern authentication mechanisms * User-based rate limiting and concurrent request management with CORS support * HMAC signature authentication for enhanced security and custom response event scripting * Sample request templates for [XML Map](./xml-map/xml-map) connector integration and workflow automation ## Overview Webhook connectors enable data to enter the CData Arc flow via HTTP POSTs and PUTs. Each Webhook connector exposes an endpoint in the application where external clients can send XML and JSON payloads. These payloads are written to an output file and passed along to the next connectors in the flow. You can specify a sample request in the Webhook connector to simplify the process of transforming data that is POSTed to the endpoint. When an XML sample is specified, and the Webhook connector is connected to an XML Map connector in the flow, the XML Map connector automatically detects the expected structure of XML files posted to the endpoint. You can then use the XML Map connector [Node Value Editor](../mapping/mapping-node-value-editor) to map this structure into a target XML structure. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Connector Details *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Webhook Endpoint** | The generated URL (based on the connector name) where the endpoint is exposed. | #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Incoming Headers** | A comma-separated list of HTTP header names to read from incoming requests and promote as message headers. | | **Enable HMAC Authentication** | Check **Enabled** to use HMAC (Hash-based Message Authentication Code) signature authentication to verify the authenticity and integrity of webhook requests. See [HMAC Authentication](#hmac-authentication) for details. | | **HMAC Signature Header** | The signature header value. By default this is `x-cdata-hmac-signature`, but you can customize it. | > [!NOTE] > > Header names in **Incoming Headers** are matched case-insensitively. Only headers listed here are promoted; headers present on the request but not declared in the setting are not surfaced as message headers. If a declared header is not present on a request, the request is still processed and the message header is left empty. ### Request Details Tab Supply an XML or JSON template representing the expected structure of incoming data. The primary benefit of specifying a sample request is when you connect the Webhook connector to an [XML Map](./xml-map/xml-map) connector in the flow. Use XML Map connectors when the API data needs to be converted into some other format, like an EDI document or a database insert. The XML Map connector detects the XML structure of the sample request and uses this as the **Source File** for the XML Map connector. Upload the XML structure representing the target format as the **Destination File**, then use the Node Value Editor to convert the source structure into the destination. ### Users Tab The **Users** tab lets you: * Create users with their associated authtokens and OAuth 2.0 credentials * Define POST and/or PUT privileges * Specify how many requests each user can make per hour * Specify how many concurrent requests are permitted You can add, edit, and delete users on this tab. See Add or Edit Users for details on each field. > [!NOTE] > > The request settings here override settings in the [Default Rate Limits](#default-rate-limits-per-user) section of the **Server** tab. ### Server Tab #### Trusted IP Addresses The following functions are available in the **Trusted IP Addresses** section:
FunctionDescription
**Add**Enter a new IP address range.
**Edit**Modify the selected IP address range.
**Delete**Deletes the selected IP address range from the list.
The following restrictions apply to this feature:
  • `localhost` cannot be modified or removed from the list.
  • Any IP addresses outside of the defined ranges are rejected.
  • Ranges are supported. For example, the entry `100.10.100.1-15` indicates that IP addresses between `100.10.100.1` and `100.10.100.15` are allowed.
  • Classless inter-domain routing (CIDR) notation is supported. For example, the entry `100.10.100.0/24` indicates that IP addresses between `100.10.100.0` and `100.10.100.255` are allowed.
  • Wildcard patterns are supported. For example, the entry `100.10.100.*` indicates that IP addresses beginning with `100.10.100` are allowed.
> [!NOTE] > > In order for clients to reach the server, a clear network path is required. In cloud environments you might need to make changes in three places: > > • The networking rules in the cloud console. > > • The firewall rules on the machine hosting the application. For example, when using an Amazon AMI, you might use an *Uncomplicated Firewall* (UFW) to allow traffic on the desired port. A common strategy in Linux environments is to forward traffic from ports lower than 1024 to a non-standard port higher than 1024, while configuring the application to use the non-standard port. This avoids permission issues associated with non-root users binding to ports lower than 1024. > > • The [Network Access](/26.4/self-hosted/en/getting-started/administration/settings/network-access) portion of the [Security](/26.4/self-hosted/en/getting-started/administration/settings/security) tab. #### Default Rate Limits (Per User) *Settings restricting the number of requests allowed, if no values have been provided on the Users tab.* | Setting | Description | | - | - | | **Max Requests Per Hour** | The limit to the number of requests a single user can issue in an hour. | | **Max Concurrent Requests** | The limit to the number of concurrent requests a user can issue. | #### Cross-Origin Resource Sharing (CORS) *Settings governing the use of CORS to serve cross-origin resources.*
SettingDescription
**Enable cross-origin resource sharing (CORS)** Whether or not CORS is enabled. The rest of these options are only available when you check this box.
**Allow all domains without '\*'** If enabled, domain origins are not restricted to a specific list.
**Access-Control-Allow-Origin** The comma-delimited list of domain origins to allow. Included as an HTTP response header.
**Access-Control-Allow-Credentials** Whether or not user credentials such as cookies are allowed in cross-origin requests. Included as an HTTP response header.
**Access-Control-Allow-Methods** The comma-delimited list of methods to allow. Included as an HTTP response header.
**Access-Control-Allow-Headers** The comma-delimited list of headers to allow. Included as an HTTP response header.
**Access-Control-Max-Age** The maximum duration (in seconds) that Access-Control response header values can be cached.
#### Advanced Settings | Setting | Description | | - | - | | **Authtoken in URL** | Check this to allow the connector to pass the authtoken in query string parameters for authentication. See [Using Auth Tokens as Query String Parameters](#using-auth-tokens-as-query-string-parameters) for more information. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## HMAC Authentication HMAC (Hash-based Message Authentication Code) signature authentication is a cryptographic method that verifies the authenticity and integrity of webhook requests. It uses a shared secret key to generate a unique signature for each request, ensuring: * Request Authenticity: Confirms the request originated from a trusted source * Data Integrity: Verifies that the request payload hasn't been tampered with during transmission * Replay Attack Prevention: Protects against malicious reuse of intercepted requests This provides significantly stronger security compared to traditional authentication methods like basic authentication or API keys, which can be more easily compromised. Users with Professional or Enterprise [licenses](../getting-started/administration/settings/license-settings) can enable HMAC signature authentication for incoming webhook requests. ### Configuration > [!WARNING] > > HMAC authentication is a global setting that applies to every user authorized to access the webhook. 1. Enable HMAC Authentication 1. Check **Enable HMAC Authentication** in the webhook connector settings. 2. Optionally, customize the HMAC Signature Header value (the default is `x-cdata-hmac-signature`). ### Set HMAC Keys for Users After enabling HMAC authentication: 1. Navigate to the **Users** tab of the webhook connector. 2. Configure the HMAC key each authorized webhook user: 1. An HMAC key is automatically generated for each user. You can override the auto-generated key with a custom value. Each user must use their assigned HMAC key to generate valid signatures for their webhook requests. ## Auth Token Authentication Users can access Webhook resources by providing authtokens with requests. Manage users and authtokens by adding or editing a user on the [Users](#users-tab) tab, and navigating to the **Authentication** tab. Before users can call the Webhook endpoint, you must also set trusted IP addresses for connections. Set these in the [Trusted IP Addresses](#trusted-ip-addresses) section of the **Server** tab. By default, all IP addresses are restricted. ### Using Auth Tokens in Basic Authentication Enter the user's authtoken as the password when using Basic Authentication. ### Using Auth tokens in the HTTP Header Add the HTTP header `x-{companyName}-authtoken` with the authtoken as part of the HTTP request. ### Using Auth Tokens as Query String Parameters To allow the connector to pass the authtoken in query string parameters, check **Allow authtoken in URL** in the [Advanced Settings](#advanced-settings) section of the **Server** tab. After enabling this feature, you can specify the authtoken as the value of the `@authtoken` parameter, which you supply as part of the HTTP form-post data or as a query parameter. ## OAuth 2.0 Authentication Users can secure Webhook resources using OAuth 2.0 authentication. Manage users and OAuth credentials by adding or editing a user on the [Users](#users-tab) tab, and navigating to the **Authentication** tab. ## Receiving Data When data is uploaded to the webhook endpoint, the body of the web request is written as an output file and passed along to the next connector in the flow. This allows for a flexible method of invoking an Arc workflow via an external API call. > [!TIP] > > Data uploaded to the endpoint is not validated in the Webhook connector, and should be validated later in the flow if necessary. ## Custom Responses Ordinarily, the Webhook connector accepts the post data with a token response that the request was accepted, but you can customize the response by using the `Response` event, where the `_request`, `_httpheaders`, `_response`, and `_message` special items are available. When specified, the connector expects the custom response to be provided through the `_response` item. You can also use the `Response` event to push custom output items using the following attributes: * Filename: The filename of the output message to pass down the flow. * Data: The data to include in the message that is passed down the flow. For binary data, use the Base64Data attribute instead. * Base64Data: The Base64-encoded data to include in the message that is passed down the flow. * HeaderNames#: A list of header names to include on the message that is passed down the flow. Use the HeaderValues attribute to specify values for these headers at the matching index. * HeaderValues#: A list of header values to include on the message that is passed down the flow. These values are used for the header names defined at the matching index in the HeaderNames list. * Logs#: A list of log entries to include in the logs for the transaction. ### Response Examples To push a file containing the body of the webhook request, with a custom filename and header, down the flow, the ArcScript in the `Response` event might look like this: ```xml theme={null} ``` For most cases, the **Incoming Headers** setting on the **Settings** tab is the simplest way to surface incoming request headers as message headers. If you need custom logic or a custom response, you can also do this in the `Response` event. To surface a header on the incoming request as a header on the message that is passed down the flow, the ArcScript in the `Response` event might look like this: ```xml theme={null} ``` With the ArcScript above in the `Response` event, a client can send a request similar to the following: ```http theme={null} POST https://localhost/connector/Webhook1/webhook.rsb HTTP/1.1 content-type: application/xml X-{siteNameShort}-Authtoken: 1s7U4w0a2P3l8v9W3l0q MyWebhookHeader: Hello World! Hello World! ``` and receive the following response: ```http theme={null} HTTP/1.1 200 OK Connection: close Date: Tue, 31 Aug 2021 19:16:13 GMT X-Frame-Options: SAMEORIGIN Content-Type: application/xml Content-Length: 81 Server: Jetty(9.4.z-SNAPSHOT) Successfully processed message with MySpecialHeader=Hello World! ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Workspace Receive Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/workspace-receive Receive Arc messages transported across different workspaces, preserving metadata for correct logging, monitoring, and processing. The Workspace Receive connector receives CData Arc messages transported across different workspaces. ## Key Capabilities * Inter-workspace message transport with metadata preservation for cross-workspace flows * Simple listener configuration paired with [Workspace Send](./workspace-send) connector * Automatic file routing and logging for workspace-to-workspace file transfers ## Overview The Workspace Receive connector ensures that message metadata persists as messages are passed from one workspace to another. Since Arc uses message metadata to track files through the flow, this helps ensure that files are logged, monitored, and processed correctly. This connector has very simple configuration because each Workspace Receive connector simply "listens" for files sent by a Workspace Send connector. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration Settings *Settings related to the basic operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # Workspace Send Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/workspace-send Transport Arc messages between workspaces with metadata preservation, supporting static and dynamic delivery modes with header-based routing. The Workspace Send connector transports CData Arc messages between workspaces. ## Key Capabilities * Inter-workspace message transport with metadata preservation for cross-workspace flows * Static and dynamic delivery modes with header-based workspace routing * Paired with [Workspace Receive](./workspace-receive) connector for seamless workspace-to-workspace communication ## Overview The Workspace Send connector ensures that message metadata persists as messages are passed from one workspace to another. Since Arc uses message metadata to track files through the flow, this helps ensure that files are logged, monitored, and processed correctly. This connector has very simple configuration because each Workspace Send connector is paired with one Workspace Receive connector. Primary configuration is simply a matter of setting the appropriate workspace and the name of the Workspace Receive Connector in that workspace. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration Settings *Settings related to the basic operation of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Destination *Settings related to the Workspace destination where files are sent.* | Setting | Description | | - | - | | **Delivery Mode** | Specifies how the destination workspace is determined for messages. Choose **Static** when all messages should be sent to the same workspace, and **Dynamic** when the destination workspace should be determined by headers on the message. | | **Workspace Id** | The workspace to send messages to. Applicable when **Delivery Mode** is set to **Static**. | | **Connector Id** | The name of the Workspace Receive connector in the target workspace where files are sent. Applicable when **Delivery Mode** is set to **Static**. | | **Workspace Id Header** | The name of the header that specifies the destination workspace for messages. Applicable when **Delivery Mode** is set to **Dynamic**. | | **Connecter Id Header** | The name of the header that specifies the destination connector for messages. Applicable when **Delivery Mode** is set to **Dynamic**. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent to the other workspace. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. # X12 Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/x12 Configuration and usage guide for the CData Arc X12 connector. X12 connectors support generating X12 files from XML and converting X12 files into XML. ## Key Capabilities * Complete bidirectional EDI X12 to XML document processing with all transaction sets * Interchange and functional group header validation and generation * Support for 997/999 functional acknowledgments with automatic routing * Advanced features like SNIP validation for HIPAA compliance ## Overview When receiving X12 documents, X12 connectors validate X12 interchange headers and convert the X12 document into XML. This is useful as a staging step, because XML is the primary format that Arc uses to manipulate data in a flow. The X12 connector automatically reads the input file to determine the appropriate X12 schema, then parses the document according to this schema. When generating X12 documents, X12 connectors convert XML into X12 document syntax and apply the appropriate X12 interchange headers. This is useful as the final step for creating an X12 document, after the XML data has been fetched and transformed elsewhere in the flow. > [!NOTE] > > Interchange header validation can be avoided by selecting *Test Data* in the **Usage Indicator** connector setting. An X12 connector can also automatically generate acknowledgments for incoming X12 documents. For more information, see [Acknowledgments](#x12-acknowledgments). ## Prerequisites Before you begin, make sure you have the correct schema installed for the document type you need to process. The X12 connector natively supports the following schema definitions: 00401, 00403 and 00501. You can find them in the `app_data/x12_schemas` directory of your Arc installation. If you need different schemas, you can download the additional schema files from the [CData website](https://arc.cdata.com/schemas/) for free. After you download and extract the zip archive, place the entire folder into an appropriately named subdirectory of the `app_data/x12_schemas` directory. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Translation Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Translation Type** | Whether the connector is converting an X12 document into XML, or XML data into an X12 document. | | **Dynamically process partners** | When enabled, the connector automatically identifies and tracks trading partner relationships from EDI transactions. When disabled, you must manually define the partner relationship. See [EDI Partners Tab](#edi-partners-tab) for more information. | #### EDI Partner | Setting | Description | | - | - | | **Partner Identifier (ISA06)** | Identifies the sending party in the X12 communication (when generating an X12 document, this should be your identifier). Only applicable when **Dynamically process partners** is unchecked. | | **Personal Identifier (ISA08)** | Identifies the receiving party in the X12 communication (when generating an X12 document, this should be your partner's identifier). Only applicable when **Dynamically process partners** is unchecked. | #### Interchange Settings *Settings related to the interchange headers of X12 documents. When generating documents, these settings are applied as interchange headers in the resulting document. When parsing documents, the interchange settings are used to validate the incoming document.* | Setting | Description | | - | - | | **Partner Id Qualifier (ISA05)** | Qualifies the **Partner Identifier** by providing context to the value. | | **Personal Id Qualifier (ISA07)** | Qualifies the **Personal Identifier** by providing context to the value. | | **Control Version Number (ISA12)** | Identifies the control version for the interchange. | | **Usage Indicator (ISA15)** | Whether the data exchanged represents test data, production data, or general information. When this is set to *Test Data*, interchange headers are *not* validated. | | **Functional Group** | Check this to automatically add the sender and receiver identifiers to the [Functional Group Settings](#functional-group-settings) on the **Advanced** tab. | #### Acknowledgments *Settings related to generating and requesting acknowledgments.* | Setting | Description | | - | - | | **TA1 Acknowledgment** | Whether to send TA1 interchange acknowledgments. | | **Functional ACK** | Whether a functional ACK should be returned (when receiving) and requested (when sending). A functional acknowledgment serves as an indication of acceptance or rejection of the received interchange and transaction set. | | **Functional ACK Type** | Check this to specify whether to use a 997 or 999 functional acknowledgment. | ### Sample Files Tab *Uploading a sample file allows for easy mapping in any connected CSV Map or XML Map connectors.* Use the **Sample Files** tab to generate a source or destination XML template for use in a [CSV Map](/26.4/self-hosted/en/connectors/map) or [XML Map](/26.4/self-hosted/en/connectors/xml-map/xml-map) connector. Click **Add Sample File**. 1. If you already have an example file that reflects the expected format and structure of your data, choose **File Upload** and browse to your file. Alternatively, select **Manually Create** to create a file directly in the application. If you choose **Manually Create**, give your sample file a name. Click **Next**. 2. If you browsed to an existing file, the **View File** tab is populated with its contents. If you chose to manually create a file, enter your sample data on this tab. Click **Next**. 3. The **Translate and Preview** tab displays a preview of your sample data converted to XML. When you are satisfied with your sample, click **Add**. ### Advanced Tab #### EDI Delimiters *Settings that specify which characters separate elements, segments, and so on.* | Setting | Description | | - | - | | **Data Element Separator** | The character that separates individual data elements in the document. | | **Component Element Separator** | The character that separates elements within a composite data structure in the document. | | **Segment Terminator** | The character that indicates the end of a segment in the document. | | **Suffix** | Appended to the **Segment Terminator** to distinguish segments. | #### Interchange Settings *Additional settings related to the X12 interchange headers.* | Setting | Description | | - | - | | **Authorization Qualifier (ISA01)** | Qualifies the **Authorization Id** setting by providing context for the value. | | **Authorization Id (ISA02)** | Identifies the authorization level associated with the interchange. | | **Security Qualifier (ISA03)** | Qualifies the **Security Password** setting by providing context for the value. | | **Security Password (ISA04)** | Represents security information about the interchange. | | **Control Standards Identifier (ISA11)** | Identifies the X12 standard. The only valid value is *U*. | #### Functional Group Settings *Settings related to the functional group headers of X12 documents. These optional identifiers can help group similar interchanges or facilitate sub-addressing in an organization.* | Setting | Description | | - | - | | **Partner Identifier (GS02)** | Identifies the sending party in the X12 communication. | | **Personal Identifier (GS03)** | Identifies the receiving party in the X12 communication. | | **Date Format (GS04)** | How the datestamp should be formatted. | | **Time Format (GS05)** | How the timestamp should be formatted. | | **Responsible Agency Code (GS07)** | Identifies the issuer of the X12 standard. | | **Identifier Code (GS08)** | Identifies the version, release, and industry of the X12 standard. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Batch Transactions** | An interchange can contain multiple transactions. When this is not checked, the connector creates a separate output file for each transaction in the interchange. When checked, the connector groups all transactions into a single output file. Only applicable when the **Translation Type** is X12 to XML. | | **Use Reference Id** | When checked, when you are translating X12 into XML, reference Ids are used to name XML elements. Only applicable when the **Translation Type** is X12 to XML. For example: `<_143>value`. If this is unchecked, the Reference designator is used to name XML elements: `value` | | **Encoding** | Specifies the character encoding (such as ASCII or UTF-8). | | **Expand Qualifier Values** | When checked, XML elements containing an EDI qualifier include child elements containing the qualifier code and value. For example: ``   `ST`   `Ship To` `` | | **Functional Acks** | By default, all functional acknowledgments (997, 999) are routed to the connector selected in the flow diagram, and XML translations are not received in the **Transactions** tab. Check this to have the translated acknowledgment also be included in the **Transactions** tab. This allows functional acknowledgments to be integrated into a destination source in addition to new EDI documents. | | **Generate Description As** | When translating X12 into XML, descriptions of the X12 segments and elements can be provided as context for the X12 data. Use this dropdown to choose whether to add this context as an XML comment or as XML attributes. | | **HIPAA Schemas** | When checked, the connector uses a separate set of EDI schemas with HIPAA constraints applied when translating and validating documents. This setting is ignored for EDI documents other than healthcare documents subject to HIPAA. | | **HIPAA 278** | Two separate schemas can be used for 278 documents, depending on whether these should be interpreted as requests or responses. When checked, the request schema is used; otherwise the response schema is used. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Nest Loops** | When checked, the connector detects EDI structures that have hierarchical relationships embedded in the EDI data, and generates XML with these hierarchical relationships represented as parent-child relationships. See [Master-Detail Hierarchy: Translating HLLoops](#master-detail-hierarchy-translating-hlloops) for more information. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **SNIP Validation** | When enabled, the connector performs the first three levels of SNIP Validation for HIPAA compliance: (1) the syntactical integrity of the document, (2) the presence of required segments and appropriate repetitions for repeated segments, and (3) the correct mathematical relationship between claim line items and claim totals. Higher levels of SNIP Validation (4+) might require a [Validate](./validate) connector or custom script. Read [this knowledgebase article](https://arc.cdata.com/kb/articles/snip-validation.rst) for more details on how to use SNIP Validation. | | **Strict Schema Validation** | Whether the connector should Ignore, Warn, or Fail when the following are detected: repeat counts above the allowed number, missing required elements or segments, invalid qualifier and code values, disallowed element lengths, and invalid element values. Choosing Disable turns off the schema validation checks. | | **Track ISA06** | Whether to add ISA06 values as tracked headers to processed messages. | | **Track ISA08** | Whether to add ISA08 values as tracked headers to processed messages. | | **Track Transaction Types** | Whether to add transaction types as tracked headers to processed messages. | | **Validate Identifiers** | Check this to ensure that the identifiers in the translated document match the identifiers in the connector's configuration. | | **Duplicate File Action** | How the connector should behave when it receives a message with a control number that the connector has seen before (a duplicate). When set to `Ignore`, the connector silently skips the duplicate and continues processing. When set to `Warn`, the connector processes the file but returns a warning. When set to `Fail`, the connector does not accept the file and returns an error. | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | | **Reset State** | EDI connectors keep track of control numbers that have been used and increment that number to ensure that future runs do not duplicate data. Use this button to reset the counter to its initial state without changing any of the configured settings. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### EDI Partners Tab By default, the **Dynamically process partners** checkbox on the connector **Settings** tab is enabled. This means that the connector automatically identifies and tracks trading partner relationships from EDI transactions. In this case, no manual partner configuration is required because the connector dynamically manages all partners up to your license limit, and notifies you when you reach the limit. The connector keeps track of the identifiers based on the files that it processes, which means that you might be able to use a single connector for all of your partners. ![Dynamically process partners setting](/26.4/images/edi_dynamically_process_partners.png) The **EDI Partners** tab is only enabled when Arc is dynamically processing partners. It is populated with the partner and personal identifiers of each relationship that the connector identifies, as well as the last processing date of a file for that partner. You can delete partners whose last processed date is more than 30 days ago in the **EDI Partners** tab by hovering over the partner and clicking the bin icon. ![EDI Partners tab](/26.4/images/edi_x12_partners_tab.png) When this setting is enabled, all partner relationships identified by the connector are automatically tracked for licensing purposes. They appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. Here you can view the partner and personal identifiers, the associated connectors, and date information. ![EDI partner licenses](/26.4/images/edi_partner_licenses.png) If you disable this setting, you must explicitly configure a partner relationship in the **Partner Identifier** and **Personal Identifier** fields on the **Settings** tab, as shown in the following image. ![Dynamically process partners disabled](/26.4/images/edi_x12_partner_config_disabled.png) Manually defined partner relationships also appear in the [Connector Usage](/26.4/self-hosted/en/getting-started/administration/settings/license-settings#connector-usage) portion of the **Settings > License** tab. ### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Resend Interval** | The interval the connector waits before resending a file that received a negative ACK. For example, if a trading partner receives the file but something is wrong with it and they send back a negative ACK, this setting specifies how long to wait before sending the file again. | | **Max Attempts (async)** | The maximum number of times the connector processes the input file when a functional ACK is requested. Success is based on the return of a functional ACK within the **Resend Interval**. If a successful functional ACK is not returned, the connector resends the file until **Max Attempts** is reached. If this is set to 0, the connector resends the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Converting Formats The following sections detail the process of converting X12 to XML and vice versa. ### X12 to XML Setting the **Translation Type** to *X12 to XML* instructs the connector to parse incoming X12 documents into XML. The connector first reads all of the header information for the Interchange and Functional Group sections of the document and validates them against the configured connector settings (unless **Usage Indicator** is set to *Test Data*). The connector then parses out the specific X12 schema used in the document and loads the schema from the `x12_schemas` folder on disk (additional schema files can be [downloaded from our website](https://arc.cdata.com/schemas/) for free). Using the schema, the connector generates XML representing the X12 structure of the document, populates the XML with the values from the document, and provides context to each value either as XML comments or as attributes in the XML elements (based on the value of **Generate Descriptions As**). To see this process with a set of test X12 documents, navigate to the **Transactions** tab of an X12 connector (with **Translation Type** set to *X12 to XML*) and select **More > Create Test Files**. X12 documents for an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting XML. Once X12 documents are converted to XML, the data can be transformed and manipulated in many ways. Commonly, X12 data needs to be stored in a database or other back-end application system. Since Arc uses XML to represent `inserts` into these back-end systems, storing the X12 data becomes a matter of mapping one XML structure onto another. This is typically done with the visual designer-driven [XML Map connector](./xml-map/xml-map). ### XML to X12 Setting the **Translation Type** to *XML to X12* instructs the connector to generate an X12 document out of an XML representation of the document. After the connector has constructed the X12 message out of the data parsed from the XML, it adds functional group and interchange headers according to the configured connector settings. To see this process with a set of test XML files, navigate to the **Transactions** tab of an X12 connector (with **Translation Type** set to *XML to X12*) and select **More > Create Test Files**. XML files representing an invoice, purchase order, purchase order acknowledgment, and shipping notice are automatically generated and placed in the **Transactions** directory. After these test files are processed by the connector, navigate to the **Transactions** tab to see the resulting X12 document. ### Print View When you view an X12 document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![X12 print preview](/26.4/images/x12_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## Use With the XML Map Connector In accordance with Arc convention, the X12 connector does one of two things: it receives XML input or generates XML output. To ensure that the input and output files have the correct XML structure, CData strongly recommends that you use the [XML Map connector](/26.4/self-hosted/en/connectors/xml-map/xml-map) as the previous step in the flow when generating outbound X12 documents, or as the following step in the flow when receiving inbound X12 documents. The interaction between the XML Map connector and the X12 connector is made simple by the **Upload Test File** feature described below. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect the X12 connector to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the X12 connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > XML Map connectors require both a source and destination template, so the remaining template must be set based on the structure the data originates from, or the structure it should be converted to. For example, if the data from the X12 document needs to be inserted into a database, the other template in the XML Map connector would be the XML model of a database insert. See the XML Map connector's [Using the Mapping Editor](/26.4/self-hosted/en/connectors/xml-map/xml-map-designer) for more information on creating the mapping between two template files. ## X12 Acknowledgments The following sections detail the three types of X12 acknowledgments (ACKs), and how Arc expects, processes, and generates ACKs. ### Interchange ACKs, 997s, and 999s A TA1 is the commonly accepted form of interchange acknowledgment. It is an indication that an interchange has taken place between the two parties, although not necessarily that any individual message has been exchanged. These serve as a receipt to the sender indicating that the EDI message was successfully received, but it does not specify whether there were any issues while processing the content of the message. The 997 (Version 4010), 997 (Version 5010), or 999 acknowledgements are considered functional ACKs: acknowledgements that an individual message (such as an invoice or purchase order) has been accepted. These are generated in response to a bidirectional agreement by both parties. ### Expecting, Processing, and Generating Acknowledgments #### Expecting Acknowledgements X12 connectors, while operating in XML to X12 mode, can be configured so that both interchange and functional acknowledgments can be requested for an interchange. When **TA1 Acknowledgment** or **Functional ACK** is checked in the [Acknowledgments](#acknowledgments) section of the Settings tab, the connector maintains a Pending ACK status for a transmission until the appropriate ACKs have been returned and processed. This means that the connector status can be used to determine whether the recipient has confirmed that they received the interchange. When EDI messages that require an acknowledgement have not received that acknowledgement, a Pending-Timeout header is added to the message. The message has a *Pending* status while it waits for its ACK. If it does not receive the ACK within the allotted time, the Pending-Timeout header is populated with the date and time when the message status changed to *Error*. When the ACK is received and the message is updated to *Success*, a new header is added with an empty string value. The following diagram shows the logical flow involved when expecting an ACK from an invoice document: ![Receive X12 ACK](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/x12-ack-receive.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=d960b1f774ebab18e5e81e3e29f360bc) In the illustration above, an X12 connector operating in XML to X12 mode generates the document to be exchanged **(1)** and holds it in a Pending ACK state while the document is transmitted to the trading partner. The trading partner processes the transmission according to their business logic and creates acknowledgments in accordance with the configured exchange parameters **(2)**. When the acknowledgments are returned, they are processed according to the information in the next section **(3)**. #### Processing Acknowledgments In a typical flow, X12 ACKs arrive at an X12 connector operating in X12 to XML mode. This X12 connector can be configured to automatically route any received acknowledgments to the X12 connector that originally generated the document being acknowledged. Routing ACKs between X12 connectors is configured visually on the [Flows](../flows/flows) canvas by dragging the gray dot at the bottom of the X12 connector in X12 to XML mode onto the X12 connector that is in XML to X12 mode. Once the X12 connector in XML to X12 mode receives the routed ACK, it pairs the ACK to the original message and changes its state from Pending ACK to Sent. #### Generating Acknowledgements When an X12 connector in X12 to XML mode receives a message and generates the corresponding XML, it can automatically generate acknowledgments for the received message. To accomplish this, check **TA1 Acknowledgment** and/or **Functional ACK expected** in the [Acknowledgments](#acknowledgments) section of the Settings tab. These acknowledgments must be routed to another X12 connector (in XML to X12 mode) to finalize the ACK. The connector to which the ACK is routed applies interchange headings and passes the ACK along to the next connector in the flow like any other X12 message. To route the ACK appropriately, drag the gray dot at the bottom edge of the X12 connector in X12 to XML mode onto the X12 connector that is in XML to X12 mode. The following diagram shows the logical flow involved when creating an ACK for a received invoice document: ![Create X12 ACK](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/x12-ack-create.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=b6f5092a1bd3f8bbe19809f50e56796c) As shown in the illustration above, after a trading partner sends a message expecting an acknowledgment **(1)**, the X12 connector in X12 to XML mode that parses the document automatically generates an ACK **(2)**. This ACK is an XML file containing the transactional information relevant for the transaction. Then, this XML acknowledgment is routed back to an X12 connector in XML to X12 mode **(3)** so that the interchange headers (and any other X12 party agreement settings) are applied before the ACK is sent back to the trading partner. ## Master-Detail Hierarchy: Translating HLLoops In EDI documents, most hierarchical relationships are represented by the order of EDI segments. Some EDI structures, like HLLoops (which are found in *Advance Shipment Notifications*, or X12 856s), do not follow this convention and instead have the hierarchical relationships embedded in the EDI element data itself. This can make it difficult to preserve these hierarchical relationships when converting the EDI data to XML. The X12 connector supports preserving hierarchical relationships in HLLoops via the **Nest Loops** setting in the [Advanced Settings](#advanced-settings) portion of the Advanced tab. When enabled, the connector parses the elements in the HLLoop segments to determine which segments "belong to" other segments in a hierarchical relationship. These hierarchical relationships are reflected in the output XML as parent-child relationships; in other words, this setting converts hierarchy implied by the EDI content into hierarchy represented by the XML structure. This section explains how hierarchy is encoded in HLLoop data, and how this hierarchy is converted into XML when **Nest Loops** is enabled. ### HLLoop Hierarchy All HL segments have two values that help establish hierarchy: * an Id value, which identifies the current HL segment (this value is stored in HL01, or the first element in an HL segment) * a parent-Id value, which identifies the current HL segment's hierarchical parent (this value is stored in HL02, or the second element in an HL segment) For example, an HL segment has an Id value of '2'. If the next HL segment should "belong to" this prior segment in a hierarchical relationship, then the next HL segment's parent-Id should also be '2'. If an HL segment's parent-Id is '0', the segment does not have a parent (it is at the top level of the hierarchy). #### HLLoop Example To further clarify the hierarchical relationships in HLLoop data, take the following EDI snippet as sample input (only the lines beginning with *HL* are relevant to determining the data hierarchy; the remaining segments are included to more closely represent actual EDI data): ```csharp theme={null} HL*1*0*S // First HL loop, the shipment loop TD3*RR*SEAU*1234567 HL*2*1*O // Second HL loop, the order loop PRF*PO111111 HL*3*2*P // Third HL loop, the first package loop MAN*UC*PACKAGE1 HL*4*3*I // Fourth HL loop, the first item loop MAN*UC*ITEM_1 HL*5*3*I // Fifth HL loop, the second item loop MAN*UC*ITEM_2 HL*6*2*P // Sixth HL loop, the second package loop MAN*UC*PACKAGE_2 HL*7*6*I // Seventh HL loop, the third item loop MAN*UC*ITEM_1 HL*8*6*I // Eighth HL loop, the fourth item loop MAN*UC*ITEM_3 ``` In this data, the second HL segment "belongs to" the first HL segment in a hierarchical relationship. This can be determined by looking at the HL01 and HL02 values for these segments: * The HL01 (Id) value for the first HL segment is '1' * The HL02 (parent-Id) value for the second HL segment is also '1' The same process of matching parent-Id values to Id values can be used to determine all of the hierarchical relationships in the data. ### Converting HLLoop Hierarchy to XML When **Nest Loops** is enabled, the X12 connector handles the conversion of HL hierarchy to XML hierarchy automatically. The connector parses the HL01 and HL02 element values from the EDI input and ensures that XML elements are appropriately nested (indented) inside the elements representing their parents. To see how the HL hierarchy is converted into XML hierarchy, the following XML output is generated from the above sample HL data when **Nest Loops** is enabled: ```xml theme={null} 10S RRSEAU1234567 21O PO111111 32P UCPACKAGE1 43I UCITEM_1 53I UCITEM_2 62P UCPACKAGE2 76I UCITEM_1 86I UCITEM_3 ``` ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## X12 Operations In addition to the [Operations](../scripting/operations/operations) provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except that they must be called through the `connector.rsc` endpoint and must include an auth token. Operations specific to the functionality of the X12 connector are listed below. ### x12Scan Scans header values from the headers of an X12 document. #### Required Parameters * **file**: The path to the X12 file. #### Optional Parameters * **format**: The file format. The default value is 'X12'. #### Output Attributes * **InterchangeSenderIdQualifier**: The interchange qualifier of the sender Id (ISA05). * **InterchangeSenderId**: The interchange sender Id (ISA06). * **InterchangeReceiverIdQualifier**: The interchange qualifier of the receiver Id (ISA07). * **InterchangeReceiverId**: The interchange receiver Id (ISA08). * **InterchangeControlNumber**: The value that uniquely identifies the EDI interchange (ISA13). * **FunctionalGroupSenderId**: The functional group sender Id (GS02). * **FunctionalGroupReceiverId**: The functional group receiver Id (GS03). * **DocumentType**: The document type or transaction set Identifier code (ST01). * **StandardVersion**: The major version of the X12 Standard used in this interchange (e.g. 00401, 00501). ## Common Errors Following is a list of common errors, their causes, and the recommended solution. ### Error: Schema file not found (\[schema major version] \[schema document type]) **Cause** When Arc parses an X12 file, it first scans the file for the EDI version and document type to determine which schema to use during parsing. The application includes an extensive set of X12 schemas in JSON format. They are stored in the `schemas` folder in the application directory (next to the `data` folder). This error indicates that the application could not find a schema file matching the major version and document type reported by the file. If the \[schema major version] value or \[schema document type] value are missing in the error message, the EDI document does not contain the major version or document type. Otherwise, this error indicates that the appropriate document schema is not included in the set of schema definitions automatically included with Arc. **Resolution** If the EDI file does not contain the schema major version or document type, contact the trading partner (or other source of the EDI document) and inform them that these values must be included in the message header. Otherwise, find the appropriate schema version from our free download page. Download the schema folder and place it in one of the following locations: 1. Inside the schema folder on disk (for example, `/Resources/Schemas/`) 2. If a schema is used by multiple connectors, CData recommends you use a folder structure like this: `/schemas/_schemas/`. In this example, `` should be `x12`. ### Error: The segment at line \[line number] with tag \[segment name] is not in a valid position in the specified schema (\[schema major version] \[schema document type]). This may indicate that the input file contains segments that are out of order. **Cause** When Arc parses an X12 file, it first scans the file for the EDI version and document type to determine which schema to use during parsing. The application includes an extensive set of X12 schemas in JSON format, stored in the `www\app_data` folder of the root installation directory. This error indicates that the parser encountered an EDI segment in the file that does not match the order of segments defined in the relevant document schema. This can indicate a few different issues: * The EDI file was generated incorrectly by the trading partner * The EDI file was generated using a different schema version than what is indicated in the file * The EDI file was generated using a custom schema **Resolution** If the EDI file was generated using a custom schema, place the custom schema file (in JSON format) in the folder where Arc reads EDI schemas. The connector searches for schemas by checking the following locations in this order: 1. Inside its folder on disk (for example, `/Resources/Schemas/`) 2. If a schema is used by multiple connectors, CData recommends you use a folder structure like this: `/schemas/_schemas/`. In this example, `` should be `x12`. If the EDI file was not generated using a custom schema, there are two approaches to resolving the discrepancy between the EDI file generated by the partner and the EDI schema used to parse it. 1. Manually edit the appropriate EDI schema file (found in the filepath above) to adjust the segment order so that it matches the partner's EDI files. The error message indicates the line in the file where the segments diverge from the schema definition. Trace through the JSON definition of segment order until the divergence is found, and re-order the segments in the JSON to match the file provided by the partner. 2. Contact the trading partner about the EDI file. Confirm that the file was generated according to the major version that is reported by the file, and explain that the order of the segments in the file is not recognized. Which approach to use depends on your familiarity with EDI segments for the particular document type (to correctly edit the JSON schema according to the partner's file) and the partner's flexibility (to adjust the EDI file in response to informing them about the error). # XML Group Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-group Group and aggregate collections of related XML records by key fields, with support for MIN, MAX, SUM, AVG, COUNT, and CONCATENATE functions. The XML Group connector processes collections of related XML records, grouping them by columns for efficient data management and analysis. ## Key Capabilities * Hierarchical XML record grouping and aggregation with key-based grouping and multiple grouping keys * Aggregate function support (MIN, MAX, SUM, AVG, COUNT, and CONCATENATE) with XPath-based identification * Preserves original record structure while adding summary and grouping elements ## Overview The XML Group connector processes collections of related XML records by aggregating, validating, and transforming structured data for seamless integration with various systems. It enables efficient handling of hierarchical XML data, ensuring consistency and interoperability across applications such as data exchange, reporting, and automated workflows. ## Connector Configuration This section contains all of the configurable connector properties. ### Group Config Tab *Settings related to configuring the group.* #### Sample File *Use the **Sample File** toggle to specify a sample file that allows the connector to provide more context about the document. See [Using Sample Files](#using-sample-files) for more information.* | Setting | Description | | - | - | | **Sample File** | Select an existing sample file, or upload a new one. | #### Loop Settings *Settings related to the XPath the connector should use to represent a single record in the input document, as well as the XML element to use to hold each group of related records.* | Setting | Description | | - | - | | **Element Name** | The name of the XML element to use for the grouping in the output file. | | **Record XPath** | Which XPath from the input should represent a single record. | #### Group Settings *Settings that define which elements in each record to use to determine the key for each group. See [Using Group Settings](#using-group-settings) for more information.* | Setting | Description | | - | - | | **Element Name** | Tells the connector what to name the XML element to use for grouping in the output file. This is usually the same name as the element in the input file that the Key XPath points to, but you can use a different name. | | **Key XPath** | The XPath from the input file of the element to use as the *key* (what you want to group by). The **Key XPath** is relative to the **Record XPath**. | | **Add Key XPath** | Click this button to define additional keys. | #### Summary Settings *Use the **Summary Settings** toggle to see the settings that define how to summarize the records in each group. See [Using Summary Settings](#using-summary-settings) for more information.* | Setting | Description | | - | - | | **Element Name** | The name of the XPath to use to summarize the records. | | **Function** | The function to use as the aggregate. Choose from MIN, MAX, SUM, AVG, COUNT, and CONCATENATE. | | **XPath** | The relative XPath to the element to aggregate. | | **Add Aggregate XPath** | Click this button to add more aggregate XPaths. | #### Order By Settings *Use the **Order By** toggle to see the settings that define how to sort the records in each group.* | Setting | Description | | - | - | | **XPath** | The relative XPath to the element to sort by. | | **Order By** | Whether to sort in ascending or descending order. | | **Add Order By XPath** | Click this button to add more XPaths to sort by. | ### Settings Tab #### Configuration *Settings related to establishing the connection.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Check this to have the connector automatically send groups when they are ready. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Using Sample Files Toggle **Sample File** on to upload a sample input XML that mimics the XML input that is going to be passed into the connector. When enabled, it provides XPath helpers for each of the XPath fields in the UI, so it becomes easier to see the document structure and select the correct XPath. You can only click eligible XML elements, based on what the XPath is representing for the connector. For example, in **Loop Settings**, you can only select parent elements because Arc expects that those elements are what represent a *record*. ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-group-loop-settings.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=4f1165963d54628f5e4be803367b3cd9) Then, in **Group Settings**, the visible elements are the child elements of the **Loop Settings** parent element because Arc expects the *key* element to be a child of the parent record. ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-group-group-settings.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=f5d04d4507609758e7e50ee4a37c156b) ## Using Group Settings Group settings define the elements in each record to use to determine the key for each group. The key is usually an `Id` value such as `CustomerID` or `InvoiceID`. You can have multiple keys, which appear as children of the element name and xpath set in [Loop Settings](#loop-settings). * **Element Name** tells the connector what to name the XML element to use for grouping in the output file. This is usually the same name as the element in the input file that the Key XPath points to, but you can use a different name. * **Key XPath** is the XPath from the input file of the element to use as the *key* (what you want to group by). The **Key XPath** is relative to the **Record XPath**. For example, in the following XML, the `` element is the key. This lets you group all occurrences of `ALPHA`, `BRAVO` and `CHARLIE` together in the output, instead of having multiples of each. ```xml theme={null} ALPHA Alpha Materials 1000.00 ALPHA Alpha Materials 2000.00 BRAVO Bravo Materials 100.00 CHARLIE Charlie Materials 500.00 CHARLIE Charlie Materials 100.00 ``` ## Using Summary Settings Summary settings control how the connector should aggregate data. You can define an XPath containing a value that you want to perform an action operator on (MIN, MAX, SUM, AVG, COUNT, CONCATENATE), and the connector then performs that operator on all records in a group. A common example is to SUM the price of each record. The following example uses the **Loop Settings** and **Group Settings** described in the previous images, along with the following **Summary Settings**: ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-group-group-settings.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=f5d04d4507609758e7e50ee4a37c156b) Using the XML in the previous section results in an output XML with a `InvoiceDetailGroup` for each key element (the `MaterialCode`). Inside each `InvoiceDetailGroup` is the `MaterialCode` key from the **Group Settings**, and the calculated `NetAmountSUM` from the **Summary Settings**. Finally, the original `InvoiceDetail` elements and their children are included: ```xml theme={null} ALPHA 3000 ALPHA Alpha Materials 1000.00 ALPHA Alpha Materials 2000.00 BRAVO 100.00 BRAVO Bravo Materials 100.00 CHARLIE 600 CHARLIE Charlie Materials 500.00 CHARLIE Charlie Materials 100.00 ``` Because you already have the associated records as nested children of the key, instead of in a flat structure, this XML is easier to work with when it goes to an [XML Map](./xml-map/xml-map) connector. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# XML Map Connector Overview Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-map/xml-map Transform XML data from one structure to another using the visual mapping editor in CData Arc. The XML Map Connector provides a visual designer-driven way to transform XML data from one structure to another. XML is the primary format that CData Arc uses to retrieve and manipulate data. Files are often converted to XML when they enter an Arc flow, and they are generated from XML when they leave the XML flow. Therefore, transforming between XML structures is critical to many use cases. ## Key Capabilities * Visual XML-to-XML transformation with drag-and-drop field mapping * AI-assisted mapping generation using supported Large Language Models (LLM) for intelligent mapping suggestions (see [AI Settings](../../getting-started/administration/settings/ai-settings) for details) * Support for complex transformations including functions, conditional logic, and virtual nodes * Advanced features like looping, filtering, and data aggregation with Foreach relationships ## Overview The XML Map connector provides a flexible and intuitive interface for transforming XML structures. First, a source template file (a file containing the XML structure for all input documents) and a destination template file (a file containing the XML structure that should be output by the connector) should be uploaded to the connector. See [Generating Template Files](./xml-map-template-files) for details. After the source and destination template files have been uploaded, the Mapping editor is populated with the source and destination XML structure. Then, elements from the source can be dragged-and-dropped onto elements in the destination to establish the mapping relationship. See [Configure the Mapping](../../mapping/mapping-config) for details. Advanced mapping features like [formatters](../../mapping/mapping-node-value-editor#formatters), [conditionals](../../mapping/mapping-condition-editor), and [custom scripting](./xml-map-advanced) are also available in the XML Map Connector. The [How-To Guides](../../how-to/how-to) include several guides for EDI mapping flows. These guides cover more than just the use of the XML Map Connector, but are good additional examples. ### Video Resources Watch this short video for a deep dive into many of the XML Map connections options. [https://www.youtube.com/embed/oh\_WITOdJ2s](https://www.youtube.com/embed/oh_WITOdJ2s) # Advanced Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-map/xml-map-advanced Use custom ArcScript in the XML Map connector for complex transformations beyond the mapping editor. The XML Map connector is designed to perform as much of the mapping as possible through the [Mapping Editor](./xml-map-designer). However there might be situations where scripting is required to handle custom use cases. ## Custom Scripts To access additional operations beyond the formatters provided in the [Node Value Editor](../../mapping/mapping-node-value-editor), click the angle brackets ![formatter options](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_anglebrackets.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=b1108961fa18a48a15c0c85d147e6960) adjacent to a destination element to open a custom script window for that element. The custom script editor supports all the features of ArcScript found in the [Scripting](../../scripting/scripting) section. As with other sections of ArcScript in Arc, the code block contains an `arc:info` section where the input parameters available to the script are defined. There is also an output element `result.text` that can be set to return the results of the custom script. The value returned in `result.text` is the value used to populate the destination element. For example, you can use a script to determine the SKU for an item based on the item's name. A simple way to accomplish this is to use a select/case statement in combination with the XPath formatter to check the `ItemName` element of the input XML: ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-map_custom_script_select.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=4286dc03695fc2016081424a80b05bdc) > [!NOTE] > > The editor validates your script as you type. If you see an Invalid Script message, you have some sort of syntax issue. ## ArcScript Operations ArcScript, including all [Operations](../../scripting/operations/operations), is fully available in the code view. For example, your source XML might contain an item ID and the SKU for that item must be retrieved from a database; in this case you can use the [dbQuery](../../scripting/operations/op-db-query) operation to look up the SKU for the ID. ## Conditional Logic ArcScript also supports performing conditional logic in a mapping template. The [arc:if](../../scripting/keyword-reference/op-arc-if) keyword is one of many keywords available to assist in performing conditional logic in templates. For example: ```xml theme={null} [paidInFull] ``` ## Foreach Loop Index When an element mapping is in a Foreach loop, the index of the current loop is always available in a custom script. The `_index` attribute is reserved for the current index of the innermost Foreach loop that contains the current element. For example, imagine mapping a `LineItem` element that exists in two Foreach loops: one loop for each order in the document, and another loop (in the first loop) for each item in an order. Referencing `[_index]` in the `LineItem` element mapping returns the number of times the inner loop (the 'item' loop) has looped so far. This value is 1-indexed. ## Map Item It can be useful to set scripting variables in one place in the mapping then reference those variables again in other parts of the mapping. This is supported by creating variables in the [Node Value Editor](../../mapping/mapping-node-value-editor) and/or by using the `_map` item described below. The `_map` item is just like any other ArcScript item, except that its scope encompasses an entire document processed by the XML Map connector. Any attributes of the `_map` item persist throughout the mapping and are only cleared when the XML Map connector finishes processing a file. For example, a mapping project might require tallying up the total cost of multiple line items in a purchase order (perhaps the mapping includes some number of *LineItemCost* elements and also a *TotalCost* element). You could use ArcScript to keep track of the running total by setting an attribute of the `_map` item like the following: ```xml theme={null} ``` The line above adds the value of the *LineItemCost* element to the current value of `_map.sum_cost` (with a default value of 0 if `_map.sum_cost` does not yet exist). If this code is included in an element in the Foreach loop that loops over all of the line items, the value of `_map.sum_cost` is be the *TotalCost* when the Foreach loop exits. Since the attributes of `_map` are preserved, this same `_map.sum_cost` value can be referenced later in the *TotalCost* element mapping. For example: ```xml theme={null} [_map.sum_cost] ``` # XML Map Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-map/xml-map-configuration All configurable properties for the XML Map connector in CData Arc. ## XML Map Tab Use this tab to upload your source and destination template files, and to build your mappings. See [Configure the Mapping](../../mapping/mapping-config) in the [Mapping](../../mapping/mapping) documentation for detailed instructions on how to use the mapping editor. ## Settings Tab ### Connector Configuration Settings related to the core operation of the connector.
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
## Advanced Tab ### AI
Setting Description
**Customize AI Prompts** Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings Advanced file processing settings. | Setting | Description | | - | - | | XML Declaration | Specify whether to include the XML declaration in generated XML files. | | Cache Content | Specify whether the parser should cache the original raw content of each XML element so it can be retrieved by `xsubtree` during mapping. This increases memory usage but allows mappings to copy the original content of subtrees verbatim. See [Copying XML Subtrees](#copying-xml-subtrees-xsubtree) for details. | | Local File Scheme | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | Processing Delay | The amount of time (in seconds) the connector waits before processing a file. | ### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Automation Tab ### Automation Settings Settings related to the automatic processing of files by the connector. | Setting | Description | | - | - | | Send | A toggle that instructs the connector to automatically send files when they are ready. | #### Performance | Setting | Description | | - | - | | Max Workers | The maximum number of worker threads consumed from the threadpool. | | Max Files | The maximum number of files sent by each thread. | ## Alerts Tab Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](../../getting-started/administration/settings/settings-page#alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window, set the minimum number of transactions you expect to be processed, use the **Every** fields to specify the time frame, indicate when the SLA should go into effect, and check the boxes for the days of the week. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. Check **Send an 'At Risk' notification** to receive notifications before violation. > [!NOTE] > > You can turn off SLA alerts during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Copying XML Subtrees (xsubtree) The [xsubtree](../../scripting/value-formatters/function-formatters#xsubtree-xpath) formatter copies the original raw XML content of a source element verbatim to the destination, preserving the full subtree structure without re-serializing it through the mapping engine. This is useful when part of a source document needs to pass through unchanged, regardless of its internal structure. To use `xsubtree`, first enable **Cache Content** on the [Advanced](#advanced-tab) tab. This instructs the parser to retain the original raw content of each XML element so that `xsubtree` can retrieve it. Enabling this setting increases memory usage proportionally to the size of the source document. Once **Cache Content** is enabled, use the `xsubtree` formatter in a [custom script](./xml-map-advanced#custom-scripts) on a destination node: ``` [xsubtree(ElementName)] ``` Where `ElementName` is the name (or relative XPath) of the source element whose subtree you want to copy. ### Example Given the following source document: ```xml theme={null} 1 2 Hello World Test ``` A mapping with a Foreach loop over `/Items/Item` and the script `[xsubtree(Sub)]` on a destination node produces the following output: ```xml theme={null} 12 HelloWorld Test ``` Each `Sub` element is copied exactly as it appears in the source, including all child elements, regardless of how they differ between iterations. > [!NOTE] > > If a specified element does not exist in the source, `xsubtree` returns an empty result. ## Macros | Macro | Description | | - | - | | ConnectorID | The Connector ID of the connector. | | Ext | The file extension of the file (e.g., `xml`, `csv`). | | Filename | The original filename of the message. | | FilenameNoExt | The original filename of the message, without the file extension. | | MessageId | The unique identifier assigned to the message. | | RegexFilename | A filename derived from a regex on the original filename. | | Header | The value from a message header, referenced by name. | | LongDate | The current date/time in long format. | | ShortDate | The current date/time in short format. | | DateFormat | The current date/time in a custom format. | | Vault | The value from a vault entry, referenced by name. | # Using the Mapping Editor Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-map/xml-map-designer Use the XML Map Mapping Editor to traverse and map source and destination XML structures. Once you have specified the source and destination [template files](./xml-map-template-files), the Mapping Editor populates with the complete document model for the source and destination XML. These models can be traversed as an XML tree. ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-map-mapping-editor.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=567e0fc88dcdc732e5e16fb4663b1558) See [Configure the Mapping](../../mapping/mapping-config) in the [Mapping](../../mapping/mapping) documentation for detailed instructions on how to use the mapping editor. # Generating Template Files Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xml-map/xml-map-template-files Choose or upload template files for source and destination XML structures in the XML Map connector. The first step in any XML mapping is to choose or upload template files representing the source and destination XML structure. ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xml-map-template-files.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=e10f1573552ea2b15ae44edc8443a95e) * *Source files* represent the XML structure of input documents. Any files processed by the connector should have a matching XML structure. Elements in input documents can be repeated in ways that differ from the source file as long as the structure of nested elements is the same. * *Destination files* represent the structure of output documents. Files produced by the connector have a matching XML structure. Elements in the output document can be repeated or omitted in ways that differ from the destination file according to the mapping; however the structure of nested elements remains the same. These samples can be generated within Arc in several ways. ## Template Files from Transformation Connectors Transformation connectors like X12, EDIFACT, and CSV automatically convert documents into XML. In addition, EDI connectors make the full file schema available to the XML Map connector. This provides an easy starting point for mapping your EDI file into the desired format, especially in cases where a single test file might not cover all the elements that could be required in the mapping. ### Upload Test File Follow these steps to generate an XML representation of an input file: 1. In the **Transactions** tab of the connector, click the **More** dropdown and click **Upload Test File**. 2. Navigate to a file on disk that you want to model as XML, select it, and click **OK**. 3. Connect an [EDI connector](/26.4/self-hosted/en/connectors/edi-connectors) to an XML Map connector in the flow. This connection can go in either direction: inbound to the XML Map connector, or outbound from the XML Map connector. The XML Map connector automatically detects the structure of the test file. Based on where you placed the EDI connector, the file appears in the **Source File** or **Destination File** drop-down menu of the XML Map connector. > [!NOTE] > > Test files are only available in the XML Map connector if a compatible connector is linked to XML Map in the flow. ### Manually Create a Test File You can also manually create test files. Simply send a sample file through the transformation connector (EDI connectors should be explicitly configured as 'EDI-to-XML' mode) to generate an XML output file. Download that output file and upload it again as the source or destination file in an XML Map connector. ## Choose a Built-In Template File When you connect an [EDI connector](../edi-connectors) that contains built-in schema files to an XML Map connector, the XML Map connector can automatically read the schema files and provide them as options in the **Source File** or **Destination File** fields on the XML Map **Settings** tab. These template files are prefixed by `connector://`. ## Template Files from Database Connectors Database connectors like the SQL Server, MySQL, and CData connectors automatically generate XML representations of their Input and Output mappings. Similar to EDI and CSV connectors, after a source or destination mapping is saved in a database connector, any connected XML Map connectors can automatically detect these XML structures. To use a database table structure as a source or destination file, follow these steps: 1. In the database connector, establish a working connection to the database. 2. Choose an action (for example, Select, or Upsert), and configure it accordingly. Optionally, you can use the **Preview** button to save a sample for mapping that includes sample data. 3. Connect an XML Map connector to the database connector in the flow. 4. Find the table structure as a source or destination file in the XML Map connector settings. ## Best Practices XML Map connectors are typically between two other connectors in an Arc flow. It is usually best to generate template files with the two connectors surrounding the XML Map connector. For example, the following flow has an XML Map connector mapping X12 files to a database insert: ![](https://mintcdn.com/cdata-arc/d_uxoDpYg8BiVilE/26.4/images/xmlmap_example_flow.png?fit=max\&auto=format\&n=d_uxoDpYg8BiVilE\&q=85\&s=91a64e8a2fa32e6e6a9ef11b6ed93d1c) In this example, you should use the X12 connector and the SQL Server connector to generate the XML template files for the XMLMap connector between them. Alternatively, choose from the lists of built-in template files. # XSD Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xsd Validate XML documents against an uploaded XSD schema definition, automatically detecting compliance errors in incoming XML files. The XSD connector provides support for XML schema definition validations. ## Key Capabilities * XML Schema Definition (XSD) validation for ensuring XML document compliance * Automatic validation of incoming XML files against uploaded XSD schema * Schema-based error detection for data quality assurance in flows ## Overview After an XSD file has been uploaded to the connector, XML files arriving at the connector are automatically validated based on the XSD definition. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Connector Settings *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **XSD Schema File** | The XSD file that defines the validations to be performed by the connector. | Use the **XSD Schema File** dropdown to select an existing file, or to upload a file or set of files. When you choose **Upload File**, the **Upload Type** dropdown has three options: * **Upload Directory**: Upload a folder containing multiple XSD files. When you choose the folder, its directory structure appears and you can choose the specific XSD file required for this connector. * **URL**: Retrieve an XSD schema directly from a remote URL. * **Manual Upload**: Upload a single file. ### Advanced Tab #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# XSLT Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/xslt Apply XSLT 1.0 stylesheets to transform XML documents, with support for the document() function and embedded scripts for advanced transformations. The XSLT connector provides support for Extensible Stylesheet Transformation (XSLT 1.0) transforms. ## Key Capabilities * XSLT 1.0 transformation support for converting XML documents based on stylesheet templates * Document function and embedded script support for advanced transformations ## Overview After an XSLT file has been uploaded to the connector, XML files arriving at the connector are automatically converted based on the XSLT definition. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Connector Settings *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **XSL Template File** | The XSLT file that defines the XML transformation to be performed by the connector. | #### Other Settings *Settings not included in the previous categories. Only visible if you use the [.NET edition](../getting-started/installation/windows-edition).* | Setting | Description | | - | - | | **Document Function** | Check this to allow use of the `document()` function while processing XSLT. | | **Embedded Scripts** | Check this to allow use of embedded scripts while processing XSLT. | ### Advanced Tab #### Advanced Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
## XSLT Operations In addition to the [Operations](../scripting/operations/operations) provided with Arc, connectors can provide operations that extend functionality into ArcScript. These connector operations can be called just like any other ArcScript operation, except for two details: 1. They must be called through the `connector.rsc` endpoint. 2. They must include an auth token. For example, calling a connector operation using both of these rules might look something like this: ```xml theme={null} ``` The operation specific to the functionality of the XSLT connector is described below. ### xsltTranslate Translates an input XML file according to a configured XSLT connector or a specified XSLT file. #### Required Parameters * **ConnectorId**: The name of the configured XSLT connector to use for translation, *OR* * **TemplateFile**: The path to the XSL template to use for translation, *OR* * **Data**: The contents of the XSL template to use for translation * **File**: The input file to translate #### Optional Parameters * **OutputEncoding**: By default, the operation outputs ASCII data. To output base64-encoded binary data instead, set this parameter to `base64`. * **OutputFile**: When this is set, output data is not written to the **Data** output attribute and is instead written to the specified file. #### Output Attributes * **Data**: The output from the translation is passed through this parameter. # Zip Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/zip Automatically compress and decompress archives in multiple formats (7zip, Bzip2, Gzip, Tar) with password protection and scheduled send support. The Zip connector supports automatic compression and decompression of archives in various formats. ## Key Capabilities * File compression and extraction with support for multiple archive formats (7zip, Bzip2, Gzip, and Tar) * Password protection and encryption capabilities for secure archives * Scheduled Send functionality to combine multiple files into single archives * Configurable compression methods and levels for optimized file sizes and processing speed ## Overview Zip connectors automatically compress or decompress incoming files depending on the **Operation** setting of the connector. In *Compress* mode, each file is zipped independently, so the number of output Zip archives is equal to the number of input files. In *Decompress* mode, all files are extracted from input archives and output as an individual file; folder structure is ignored. Zip connectors can be configured to compress and decompress archives in a variety of formats, including 7zip, Bzip2, Gzip, and Tar. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core operation of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Operation** | Whether the connector is compressing or decompressing incoming files. Trying to decompress files that are not compressed in a supported format results in an error. | #### Archive Options *Settings related to the archive being processed or created.* | Setting | Description | | - | - | | **Archive Format** | The expected format of the input archive (when decompressing), or the format of the archive that is created (when compressing). | | **Compression Method** | The specific algorithm to use when compressing. | | **Compression Level** | The level of compression to apply when compressing. Higher values allow better compression; lower values allow faster compression. A value of 0 stores the file without compression. | #### Encryption *Settings related to encrypting archives during compression, if allowed by the archive format.* | Setting | Description | | - | - | | **Encryption Algorithm** | The algorithm to use when encrypting. | | **Password** | The password required to decrypt the archive. | ### Advanced Tab #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Extract To Folder** | The folder where the connector should place files extracted from an archive. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* By default, the **Enable automation** checkbox is checked. If you uncheck it, you cannot use the other automation settings. | Setting | Description | | - | - | | **Send** | Whether messages arriving at the connector are automatically processed. | | **Scheduled Send** | Whether to process incoming messages at a scheduled interval. This allows multiple files to accumulate in the input folder, then all be zipped together in the same archive. **Note**: The following archive formats do not support compressing multiple files: Bzip, Gzip, and Zcompress. An error is generated if this setting is enabled on those archive formats. | | **Send Interval** | The interval at which to compress all available input files into the same archive. Only available if **Scheduled Send** is enabled. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Subfolder Headers The Zip connector makes use of the Subfolder header for reading from and writing to compressed files. A Subfolder header is added any time there are subfolders in the zip structure. > [!NOTE] > > When compressing files, the Subfolder header is only used when you set **Scheduled Send** on the Automation tab. For example, this image from the **Message Details** page for a log entry shows the path to the subfolder in the **Other Headers** section. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/subfolder_header.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=a2f2efa64c1953c055577e0b3d97b892) When decompressing a file: * The message for each file that is in a subfolder in the compressed file contains a Subfolder header in its metadata. * This header displays the path to the subfolder in the zip folder structure. * This subfolder header is supported by other connectors that support a subfolder header in Send operations. When compressing files, the connector looks for a Subfolder header on messages that it processes to determine where to place the files inside the compressed structure. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# ZUGFeRD Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/zugferd Configuration and usage guide for the CData Arc ZUGFeRD connector. The ZUGFeRD connector generates ZUGFeRD files from PDF and XML invoice files, or extracts invoice XML files from ZUGFeRD files. ## Key Capabilities * German e-invoicing standard combining PDF and XML with bidirectional processing (extract XML or generate ZUGFeRD files) * Support for versions 1.0, 2.0, 2.1 with multiple conformance levels (MINIMUM, BASIC, EN 16931, EXTENDED, and XRECHNUNG) * PDF/A-3 format support for embedding machine-readable XML in human-readable PDFs with Batch Group processing ## Overview The ZUGFeRD connector facilitates the exchange of electronic invoices by integrating the ZUGFeRD (Zentraler User Guide des Forums elektronische Rechnung Deutschland) standard, which combines human-readable and machine-readable data in a single document. It is based on the PDF/A-3 standard, which allows for the embedding of XML data in a PDF file. This hybrid format ensures that the invoice can be easily read by humans while also being processed automatically by computer systems. The ZUGFeRD connector allows you to interact with ZUGFeRD files in two ways: * [Parsing a ZUGFeRD file](#parsing-a-zugferd-file) * [Generating a ZUGFeRD file](#generating-a-zugferd-file) The connector determines which to do based on the type of input provided. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to establishing the connection.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Version** | The ZUGFeRD version to use in the generated ZUGFeRD file. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Conformance Level** | The conformance level to use in the generated ZUGFeRD file. Options are: BASIC, EXTENDED, MINIMUM, BASIC WL, EN 16931, XRECHNUNG. | | **Invoice Description** | The free text description to use in the generated ZUGFeRD file. | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Check this to have the connector automatically send files when they are ready. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Parsing a ZUGFeRD file The connector expects to receive a PDF file as input, which it reads to extract the attached XML to send down the flow. ### Handling Multiple Attachments You might receive PDF files containing multiple attachments. By default, the connector passes all the attachments down the flow, adding headers to each message: * ZUGFeRD-File-Index: The attachment number for each attachment * ZUGFeRD-File-Count: The total number of attachments on the email * ZUGFeRD-File-Name: The original PDF file name * ZUGFeRD-File-Description: The description of invoice XML and attachments * ZUGFeRD-File-Parameters: Other file parameters of invoice XML and attachments * ZUGFeRD-File-Relationship: The file relationship of invoice xml and attachments. Invoice XML is usually "Alternative", and attachments are "Data", "Supplement", and so on. * ZUGFeRD-File-Subtype: The file subtype of invoice xml and attachments. This is usually the MIME-Type of the file, such as text\xml or application\pdf. ## Generating a ZUGFeRD file The connector expects a Batch Group containing exactly one PDF Batch Message and exactly one XML Batch Message. Use the [Batch Create](./batch-create) connector to create these Batch Groups before sending them to the ZUGFeRD connector in your flow. If you include more than one PDF or XML Batch Message, or if either is missing in the Batch Group, an error is thrown. All of the ZUGFeRD specific settings in the UI (Version, Conformance Level, and Invoice Description) are only used to generate a ZUGFeRD file. Parsing does not involve these settings at all. * **Version**: Controls the ZUGFeRD version used when generating files. Arc supports versions 1.0, 2.0, and 2.1. * **Conformance Level**: The ZUGFeRD standard primarily includes the following conformance levels: * MINIMUM * Contains the most basic invoice information, such as invoice number, date, and amount. * Data is embedded in PDF/A-3 format, facilitating human readability and simple system processing. * BASIC * Builds upon the MINIMUM level by including more structured data, such as buyer and seller details, tax information, and payment terms. * Suitable for scenarios requiring moderate automation. * BASIC WL * Similar to BASIC but tailored for specific use cases, often including additional fields for certain industries or regions. * Enhances interoperability in specialized contexts. * EN 16931 (COMFORT) * Complies with the EU standard EN 16931, providing complete structured data for highly automated processing. * Ideal for complex business processes and system integration. * EXTENDED * Expands on EN 16931 by allowing the inclusion of industry- or company-specific additional information. * Designed for businesses with unique requirements. * XRECHNUNG * A German-specific profile based on ZUGFeRD, aligned with the XRechnung standard for public sector invoicing. * Ensures compliance with German public procurement regulations. * **Invoice Description**: PDF readers surface an optional description for attachments on PDFs. This setting controls what description is used for the XML attachment. ### Generating a Sample File To generate a sample file that you can use for testing, navigate to the connector's **Transactions** tab and click the **Generate sample file** ![Generate sample file](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/generate_test_file.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=887986b866be0d83f4596cc7155565e8) button. This creates a PDF file in the Send folder. When the connector processes the PDF, it generates an XML output file with the extracted invoice data from the PDF. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Adobe Commerce Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce Connect Arc to Adobe Commerce for data integration and synchronization The Adobe Commerce connector allows you to integrate Adobe Commerce into your data flow by pushing or pulling data from Adobe Commerce. ## Key Capabilities * Complete e-commerce data integration with configurable authentication schemes * Support for both basic authentication and token-based access * Store-specific data access with configurable store codes * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Adobe Commerce. ## Establish a Connection To allow Arc to use data from Adobe Commerce, you must first establish a connection to Adobe Commerce. There are two ways to establish this connection: * Add a Adobe Commerce connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Adobe Commerce**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Adobe Commerce can be reused across multiple Adobe Commerce connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Adobe Commerce. * **Auth Scheme** — The authorization scheme (basic or token) to use for the connection. * **URL** — The Adobe Commerce URL to connect to. * **User** — (Basic auth scheme only) The Adobe Commerce username to use for logging in. * **Password** — (Basic auth scheme only) The password for the user entered above. * **Access Token** — (Token auth scheme only) The access token for establishing the connection. * **Store Code** — The code of the store to connect to. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Adobe Commerce with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Adobe Commerce, you must choose the action that the Adobe Commerce connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./adobe-commerce-upsert) | Inserts or updates Adobe Commerce data. By default, if a record already exists in Adobe Commerce, an update is performed on the existing data in Adobe Commerce using the values provided from the input. | End | | [Lookup](./adobe-commerce-lookup) | Retrieves values from Adobe Commerce and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Adobe Commerce. It should be formatted as a SQL query against the Adobe Commerce tables. | Middle | | [Lookup Stored Procedure](./adobe-commerce-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./adobe-commerce-select) | Retrieves data from Adobe Commerce and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./adobe-commerce-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./adobe-commerce-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Adobe Commerce Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-ac Advanced configuration options for the Adobe Commerce connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Adobe Commerce connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Adobe Commerce connector models Adobe Commerce tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Adobe Commerce values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Adobe Commerce connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Adobe Commerce. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Adobe Commerce connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Adobe Commerce connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Adobe Commerce. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Adobe Commerce, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Adobe Commerce. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Adobe Commerce connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Adobe Commerce Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Adobe Commerce connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Adobe Commerce connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Adobe Commerce Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-lookup Retrieve values from Adobe Commerce and insert them into an Arc message in the flow. The Lookup action retrieves values from Adobe Commerce and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Adobe Commerce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Adobe Commerce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Adobe Commerce connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Adobe Commerce connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Adobe Commerce tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Adobe Commerce connector can perform Lookup queries against a local cache instead of querying Adobe Commerce directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Adobe Commerce on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Adobe Commerce directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Adobe Commerce.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Adobe Commerce Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-lsp Use a stored procedure to retrieve values from Adobe Commerce and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Adobe Commerce connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Adobe Commerce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Adobe Commerce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Adobe Commerce Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-select Retrieve data from Adobe Commerce and bring it into Arc. The Select action retrieves data from Adobe Commerce and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Adobe Commerce. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Adobe Commerce, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Adobe Commerce. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Adobe Commerce relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Adobe Commerce supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Adobe Commerce connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Adobe Commerce Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Adobe Commerce\_Select** connector pulls data from a Adobe Commerce connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Adobe Commerce\_Select** then passes down the flow from **CSV1**.
# Adobe Commerce Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-upsert Insert or update Adobe Commerce data from an Arc message in the flow. The Upsert action updates or inserts Adobe Commerce data. By default, if a record already exists in Adobe Commerce, an update is performed on the existing data in Adobe Commerce using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Adobe Commerce Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Adobe Commerce. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/adobe-commerce/adobe-commerce-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Adobe Commerce connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Adobe Commerce Upsert connector at the end: ![](/26.4/images/adobecommerce_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Adobe Commerce\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Adobe Commerce.
# Amazon Marketplace Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace Connect Arc to Amazon Marketplace for data integration and synchronization The Amazon Marketplace connector allows you to integrate Amazon Marketplace into your data flow by pushing or pulling data from Amazon Marketplace. > [!NOTE] > > Because the Amazon Marketplace connector requires that embedded credentials rotate every six months, make sure your firewall does not block `oa.cdata.com`. ## Key Capabilities * OAuth-based integration with both VendorSchema and SellerCentral schemas * Marketplace-specific operations for different selling regions * Specialized actions for order management, shipment confirmation, and inventory updates * Support for vendor payments and acknowledgment processing * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Amazon Marketplace. ## Establish a Connection To allow Arc to use data from Amazon Marketplace, you must first establish a connection to Amazon Marketplace. There are two ways to establish this connection: * Add a Amazon Marketplace connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Amazon Marketplace**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Amazon Marketplace can be reused across multiple Amazon Marketplace connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter a name for the connection. 2. Choose the VendorSchema or SellerCentral **Schema**. 3. Choose the **Marketplace** region that you are registered to sell in. 4. Provide the **Seller Id** or merchant identifier you received when you created the account. 5. Set **Include Reports** to **True** to expose existing reports in the SellerCentral and VendorCentral schemas as views. 6. Click **Connect**. 7. A new window opens for logging into Amazon Marketplace. Enter your credentials and log in. 8. If prompted, grant permissions for Arc to connect to Amazon Marketplace. You then return to Arc. 9. Click **Add Connection** to finalize the connection. 10. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 11. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Amazon Marketplace, you must choose the action that the Amazon Marketplace connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./amazon-marketplace-upsert) | Inserts or updates Amazon Marketplace data. By default, if a record already exists in Amazon Marketplace, an update is performed on the existing data in Amazon Marketplace using the values provided from the input. | End | | [Lookup](./amazon-marketplace-lookup) | Retrieves values from Amazon Marketplace and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Amazon Marketplace. It should be formatted as a SQL query against the Amazon Marketplace tables. | Middle | | [Lookup Stored Procedure](./amazon-marketplace-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./amazon-marketplace-select) | Retrieves data from Amazon Marketplace and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./amazon-marketplace-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Confirm Shipment](./amazon-marketplace-confirm-shipment) | Updates the shipment confirmation status for an order. | End | | [Submit Vendor Order Acknowledgement](./amazon-marketplace-submit-vendor-order-acknowledgement) | Submits acknowledgements for one purchase order. | End | | [Submit Vendor Shipment Confirmations](./amazon-marketplace-submit-vendor-shipment-confirmations) | Submits shipment confirmations for vendor orders. | End | | [Submit Vendor Payments](./amazon-marketplace-submit-vendor-payments) | Submits new invoices to Amazon Marketplace. | End | | [Update Inventory](./amazon-marketplace-update-inventory) | Updates inventory for specified items. | End | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./amazon-marketplace-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Amazon Marketplace Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-ac Reference for advanced settings available on the Amazon Marketplace connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the Amazon Marketplace connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Amazon Marketplace connector models Amazon Marketplace tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Amazon Marketplace values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Amazon Marketplace connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Amazon Marketplace. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Amazon Marketplace connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Amazon Marketplace connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Amazon Marketplace. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Amazon Marketplace, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Amazon Marketplace. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Amazon Marketplace connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Amazon Marketplace Confirm Shipment Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-confirm-shipment Update the shipment confirmation status for an order in Amazon Marketplace. The Confirm Shipment action updates the shipment confirmation status for an order in Amazon Marketplace. This action belongs at the end of a flow. ## XML Mapping You must connect an [XML Map](../xml-map/xml-map) connector to the Amazon Marketplace connector and map the shipment confirmation values. In the example below, the order information appears in the **Source** column, and the Amazon Marketplace shipment parameters appear in the **Destination** column. ![Amazon Marketplace Confirm Shipment XML Mapping](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_confirmshipment_mapping.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=47e3c196e3d3f92d4a8b3a3d3f932616) ## Connector Configuration In the configuration pane for the Amazon Marketplace connector, enter a comma-separated list of **Marketplace Ids** for the items that you want to update. ![Amazon Marketplace Confirm Shipment Marketplace Ids](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_confirmshipment_ids.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=2fe11469914675cc22440ae6ed917588) ## Input Parameters The following table lists the input parameters that you can map with the XML Map connector interface. | Name | Data Type | Required? | Description | | - | :-: | :-: | - | | AmazonOrderId | String | **True** | An Amazon-defined order identifier, in 3-7-7 format. | | MarketplaceId | String | **True** | The unobfuscated marketplace identifier. | | CodCollectionMethod | String | False | The cod collection method, supported in JP only. | | PackageReferenceId | String | **True** | A seller-supplied identifier that uniquely identifies a package within the scope of an order. Only positive numeric values are supported. | | CarrierCode | String | False | The code of the carrier. | | CarrierName | String | False | The name of the carrier. | | ShippingMethod | String | False | The order shipping method. | | TrackingNumber | String | **True** | The order tracking number. | | ShipDate | Datetime | **True** | The order shipping date. | | ShipFromSupplySourceId | String | False | The supply source id of the order. | | OrderItems | String | True | The list of order items, quantities and transparency codes to be updated. This aggregate field is automatically expanded in XML Map to reveal the available fields in the aggregate. | # Amazon Marketplace Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-esp Use a stored procedure to process Amazon Marketplace data and pass the result down an Arc flow. The Execute Stored Procedures action treats data coming into the Amazon Marketplace connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Amazon Marketplace connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Amazon Marketplace Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-lookup Retrieve values from Amazon Marketplace and insert them into an Arc message in the flow. The Lookup action retrieves values from Amazon Marketplace and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Amazon Marketplace connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Amazon Marketplace connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Amazon Marketplace connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Amazon Marketplace connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Amazon Marketplace tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Amazon Marketplace connector can perform Lookup queries against a local cache instead of querying Amazon Marketplace directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Amazon Marketplace on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Amazon Marketplace directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Amazon Marketplace.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Amazon Marketplace Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-lsp Use a stored procedure to retrieve Amazon Marketplace data and insert it into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Amazon Marketplace connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Amazon Marketplace connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Amazon Marketplace connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Amazon Marketplace Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-select Retrieve data from Amazon Marketplace and bring it into an Arc flow. The Select action retrieves data from Amazon Marketplace and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Amazon Marketplace. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Amazon Marketplace, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Amazon Marketplace. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Amazon Marketplace relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Amazon Marketplace supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Amazon Marketplace connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Amazon Marketplace Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Amazon Marketplace\_Select** connector pulls data from a Amazon Marketplace connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Amazon Marketplace\_Select** then passes down the flow from **CSV1**.
# Amazon Marketplace Submit Vendor Order Acknowledgement Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-submit-vendor-order-acknowledgement Submit acknowledgements for one purchase order in Amazon Marketplace. The Submit Vendor Order Acknowledgement action submits acknowledgements for one purchase order in Amazon Marketplace. This action belongs at the end of a flow. User input for this action is provided solely through the input XML. ## XML Mapping You must connect an [XML Map](../xml-map/xml-map) connector to the Amazon Marketplace connector to map the required input parameters, as shown in the image below. ![Amazon Marketplace Submit Vendor Order Acknowledgement XML Mapping](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_marketplace_SVOA.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=83020ce36335210f151b82d152aa8ef7) ## Input Parameters The following table lists the input parameters that you can map with the XML Map connector interface. | Name | Data Type | Required? | Description | | - | :-: | :-: | - | | PurchaseOrderNumber | String | **True** | The purchase-order number. | | SellerPartyId | String | **True** | The assigned identification for the party (for example, a warehouse code or vendor code). | | SellerTaxRegistrationNumber | String | **False** | The tax registration number for the entity (for example, the VAT ID). | | SellerTaxRegistrationType | String | **False** | The tax registration type for the entity. The allowed values are **VAT** (the value-added tax) and **GST.*x*** (the goods-and-services tax) only. | | SellerAddressAddressLine1 | String | **False** | The first line of the address. | | SellerAddressAddressLine2 | String | **False** | Additional address information (if required). | | SellerAddressAddressLine3 | String | **False** | Additional address information (if required). | | SellerAddressCity | String | **False** | The city where the person, business, or institution is located. | | SellerAddressCountryCode | String | **False** | The two-digit country code in ISO 3166-1 alpha-2 format. | | SellerAddressCounty | String | **False** | The county where the person, business, or institution is located. | | SellerAddressDistrict | String | **False** | The district where the person, business, or institution is located. | | SellerAddressName | String | **False** | The name of the address for the person, business, or institution. | | SellerAddressPhone | String | **False** | The phone number for the person, business, or institution that is located at that address. | | SellerAddressPostalCode | String | **False** | The postal code for the address. This code consists of a series of letters, digits, or both. | | SellerAddressStateOrRegion | String | **False** | The state or region where the person, business, or institution is located. | | AcknowledgementDate | Datetime | **False** | The date and time when the purchase order is acknowledged, in ISO 8601 date/time format. | | OrderAcknowledegment/ ItemSequenceNumber | String | **False** | The line-item sequence number for the item. | | OrderAcknowledegment/ AmazonProductIdentifier | String | **False** | The Amazon Standard Identification Number (ASIN) of an item. | | OrderAcknowledegment/ VendorProductIdentifier | String | **False** | The vendor-selected product identification of the item. | | OrderAcknowledegment/ OrderedAmount | Integer | **True** | The ordered quantity. This value should not be zero. | | OrderAcknowledegment/ OrderedUnit | String | **True** | The unit of measure for the ordered quantity. | | OrderAcknowledegment/ OrderedUnitSize | Integer | **True** | A case size (if product is ordered by the case). | | OrderAcknowledegment/ NetCostCurrencyCode | String | **False** | The three-digit currency code in ISO 4217 format. | | OrderAcknowledegment/ NetCostAmount | String | **False** | The net cost as a decimal number with no loss of precision. | | OrderAcknowledegment/ ListPriceCurrencyCode | String | **False** | Three-digit currency code in ISO 4217 format. | | OrderAcknowledegment/ ListPriceAmount | String | **False** | The list price as a decimal number with no loss of precision. | | OrderAcknowledegment/ DiscountMultiplier | String | **False** | A discount multiplier that should be applied to the price if a vendor sells books with a list price. | | OrderItemAcknowledgement/ AcknowledgementCode | String | **True** | An indicator of the acknowledgement code. | | OrderItemAcknowledgement/ AcknowledgedAmount | Integer | **True** | The ordered quantity. This value should not be zero. | | OrderItemAcknowledgement/ AcknowledgedUnit | String | **True** | The unit of measure for the ordered quantity. | | OrderItemAcknowledgement/ AcknowledgedUnitSize | Integer | **False** | The case size (if the product is ordered by the case). | | OrderItemAcknowledgement/ ScheduledShipDate | Datetime | **False** | The estimated ship date per line item. The date must be in ISO 8601 date/time format. | | OrderItemAcknowledgement/ ScheduledDeliveryDate | Datetime | **False** | The estimated delivery date per line item. The date must be in ISO 8601 date-and-time format. | | OrderItemAcknowledgement/ RejectionReason | String | **False** | The unit of measure for the ordered quantity. | # Amazon Marketplace Submit Vendor Payments Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-submit-vendor-payments Submit new invoices to Amazon Marketplace. The Submit Vendor Payments action submits new invoices to Amazon Marketplace. This action belongs at the end of a flow. User input for this action is provided solely through the input XML. ## XML Mapping You must connect an [XML Map](../xml-map/xml-map) connector to the Amazon Marketplace connector to map the required input parameters, as shown in the image below. ![Amazon Marketplace Submit Vendor Payments XML Mapping](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_marketplace_SVP.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=aea85651f8b71e35c57edff6b22cd4d3) ## Input Parameters The following table lists the input parameters that you can map with the XML Map connector interface. | Name | Data Type | Required? | Description | | - | :-: | :-: | - | | Invoice Type | String | **True** | The type of invoice. Allowed values are **Invoice** and **CreditNote**. | | Id | String | **True** | A unique number that relates to the charge details that are defined later in this table. This Id is the invoice number (if the document type is **Invoice**) or the credit-note number (if the document type is **CreditNote**). | | ReferenceNumber | String | **False** | An additional unique reference number that is used for regulatory or other purposes. | | Date | Datetime | **True** | A date and time in ISO 8601 format. | | TotalAmount | String | **True** | The total amount of the invoice as a decimal number with no loss of precision. | | TotalCurrencyCode | String | **True** | A three-digit currency code in ISO 4217 format. | | PaymentType | String | **False** | The payment term type for the invoice. Allowed values are **Basic**, **EndOfMonth**, **FixedDate**, **Proximo**, **PaymentDueUponReceiptOfInvoice**, **LetterofCredit**. | | PaymentDueDays | Integer | **False** | The number of calendar days from the base date (invoice date) until the total amount on the invoice is due. | | DiscountPercent | String | **False** | The discount percentages as a decimal number with no loss of precision. | | DiscountDueDays | Integer | **False** | The number of calendar days from the base date (invoice date) until the discount is no longer valid. | | RemitToPartyId | String | **True** | The assigned identification for the party. | | RemitToPartyAddress | Aggregate | **False** | The physical address of the person, business, or institution. | | RemitToPartyAddress/ Name | String | **True** | The name of the person, business, or institution at that address. | | RemitToPartyAddress/ AddressLine1 | String | **True** | The first line of the person, business, or institution's street address. | | RemitToPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | RemitToPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | RemitToPartyAddress/ City | String | **False** | The city where the person, business, or institution is located. | | RemitToPartyAddress/ County | String | **False** | The county where the person, business, or institution is located. | | RemitToPartyAddress/ District | String | **False** | The district where the person, business, or institution is located. | | RemitToPartyAddress/ StateOrRegion | String | **False** | The state or region where the person, business, or institution is located. | | RemitToPartyAddress/ PostalOrZipCode | String | **False** | The postal or ZIP code for the address of the person, business, or institution. This code consists of a series of letters, digits, or both, sometimes including spaces or punctuation. | | RemitToPartyAddress/ CountryCode | String | **True** | The two-digit country code in ISO 3166-1 alpha-2 format. | | RemitToPartyAddress/ Phone | String | **False** | The phone number of the person, business, or institution that is located at that address. | | RemitToParty TaxDetails | Aggregate | **False** | The tax registration details of the party. | | RemitToParty TaxDetails/ RegistrationType | String | **True** | The tax registration type for the entity. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | RemitToParty TaxDetails/ RegistrationNumber | String | **True** | The tax registration number for the entity (for example, the VAT Id). | | ShipToPartyID | String | **False** | The assigned identification for the party. | | ShipToPartyAddress | Aggregate | **False** | The physical shipping address of the person, business, or institution. | | ShipToPartyAddress/ Name | String | **True** | The name of the person, business, or institution at the shipping address. | | ShipToPartyAddress/ AddressLine1 | String | **True** | The first line of the person, business, or institution's street address. | | ShipToPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | ShipToPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | ShipToPartyAddress/ City | String | **False** | The city where the person, business, or institution is located. | | ShipToPartyAddress/ County | String | **False** | The county where the person, business, or institution is located. | | ShipToPartyAddress/ District | String | **False** | The district where the person, business, or institution is located. | | ShipToPartyAddress/ StateOrRegion | String | **False** | The state or region where the person, business, or institution is located. | | ShipToPartyAddress/ PostalOrZipCode | String | **False** | The postal or ZIP code for the address of the person, business, or institution. This code consists of a series of letters, digits, or both, sometimes including spaces or punctuation. | | ShipToPartyAddress/ CountryCode | String | **True** | The two-digit country code in ISO 3166-1 alpha-2 format. | | ShipToPartyAddress/ Phone | String | **False** | The phone number of the person, business, or institution that is located at the shipping address. | | ShipToParty TaxDetails | Aggregate | **False** | The tax registration details of the party. | | ShipToParty TaxDetails/ RegistrationType | String | **True** | The tax registration type for the party. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | ShipToParty TaxDetails/ RegistrationNumber | String | **True** | The tax registration number for the entity (for example, the VAT Id). | | ShipFromPartyId | String | **False** | The assigned identification for the party. | | ShipFromPartyAddress | Aggregate | **False** | The physical address of the shipper. | | ShipFromPartyAddress/ Name | String | **True** | The name of the shipper. | | ShipFromPartyAddress/ AddressLine1 | String | **True** | The first line of the shipper's street address. | | ShipFromPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | ShipFromPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | ShipFromPartyAddress/ City | String | **False** | The city where the shipper is located. | | ShipFromPartyAddress/ County | String | **False** | The county where the shipper is located. | | ShipFromPartyAddress/ District | String | **False** | The district where the shipper is located. | | ShipFromPartyAddress/ StateOrRegion | String | **False** | The state or region where the shipper is located. | | ShipFromPartyAddress/ PostalOrZipCode | String | **False** | The postal or ZIP code for the shipper's address. This code consists of a series of letters, digits, or both, sometimes including spaces or punctuation. | | ShipFromPartyAddress/ CountryCode | String | **True** | The two-digit country code in ISO 3166-1 alpha-2 format. | | ShipFromPartyAddress/ Phone | String | **False** | The shipper's phone number. | | ShipFromParty TaxDetails | Aggregate | **False** | The tax registration details of the party. | | ShipFromParty TaxDetails/ RegistrationType | String | **True** | The tax registration type for the party. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | ShipFromParty TaxDetails/ RegistrationNumber | String | **True** | The tax registration number for the party (for example, the VAT Id). | | BillToPartyId | String | **False** | The assigned identification for the party. | | BillToPartyAddress | Aggregate | **False** | The physical address of the party that is to be billed. | | BillToPartyAddress/ Name | String | **True** | The name of the party that is to be billed. | | BillToPartyAddress/ AddressLine1 | String | **True** | The first line of the party's street address. | | BillToPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | BillToPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | BillToPartyAddress/ City | String | **False** | The city where the party that is to be billed is located. | | BillToPartyAddress/ County | String | **False** | The county where the party that is to be billed is located. | | BillToPartyAddress/ District | String | **False** | The district where the party that is to be billed is located. | | BillToPartyAddress/ StateOrRegion | String | **False** | The state or region where the party that is to be billed is located. | | BillToPartyAddress/ PostalOrZipCode | String | **False** | The postal or ZIP code for the address of the party that is to be billed. This code consists of a series of letters, digits, or both, sometimes including spaces or punctuation. | | BillToPartyAddress/ CountryCode | String | **True** | The two-digit country code in ISO 3166-1 alpha-2 format. | | BillToPartyAddress/ Phone | String | **False** | The phone number for the party that is to be billed. | | BillToParty TaxDetails | Aggregate | **False** | The tax registration details of the party that is to be billed. | | BillToParty TaxDetails/ RegistrationType | String | **True** | The tax registration type for the party that is to be billed. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | BillToParty TaxDetails/ RegistrationNumber | String | **True** | The tax registration number for the party that is to be billed (for example, the VAT Id). | | TaxDetails | Aggregate | **False** | The total tax-amount details for all line items. | | TaxDetails/ Type | String | **True** | The type of the tax that is applied. Allowed values are **CGST**, **SGST**, **CESS**, **UTGST**, **IGST**, **MwSt**, **PST**, **TVA**, **VAT**, **GST**, **ST**, **Consumption**, **MutuallyDefined**, and **DomesticVAT**. | | TaxDetails/ TaxRate | String | **False** | The tax rate as a decimal number with no loss of precision. | | TaxDetails/ TaxAmount | String | **True** | The tax amount as a decimal number with no loss of precision. | | TaxDetails/ TaxCurrencyCode | String | **True** | The three-digit currency code in ISO 4217 format. | | TaxDetails/ TaxableAmount | String | **False** | The taxable amount as a decimal number with no loss of precision. | | TaxDetails/ TaxableCurrencyCode | String | **False** | The three-digit currency code in ISO 4217 format. | | AdditionalDetails | Aggregate | **False** | Additional details provided by the selling party, for tax related or other purposes. | | AdditionalDetails/ Type | String | **True** | The three-digit currency code in ISO 4217 format. | | AdditionalDetails/ Detail | String | **True** | The details of the additional information that is provided by the selling party. | | AdditionalDetails/ LanguageCode | String | **False** | The language code of the additional-information details. | | ChargeDetails | Aggregate | **False** | The details for the total charge amount for all line items. | | ChargeDetails/ Type | String | **True** | The three-digit currency code in ISO 4217 format. | | ChargeDetails/ Description | String | **False** | A description of the charge. | | ChargeDetails/ ChargeAmount | String | **True** | The charge amount as a decimal number with no loss of precision. | | ChargeDetails/ ChargeCurrencyCode | String | **True** | The three-digit currency code in ISO 4217 format. | | ChargeDetails/ TaxDetails | Aggregate | **True** | The details for the tax amount that is applied on this charge. | | AllowanceDetails | Aggregate | **False** | The details for the total allowance amount for all line items. | | AllowanceDetails/ Type | String | **True** | The type of allowance that is applied. Allowed values are **Discount**, **DiscountIncentive**, **Defective**, **Promotional**, **UnsaleableMerchandise**, and **Special**. | | AllowanceDetails/ Description | String | **False** | A description of the allowance. | | AllowanceDetails/ AllowanceAmount | String | **True** | The allowance amount as a decimal number with no loss of precision. | | AllowanceDetails/ AllowanceCurrencyCode | String | **True** | The three-digit currency code in ISO 4217 format. | | AllowanceDetails/ TaxDetails | Aggregate | **False** | The details for the tax amount that is applied on this allowance. | | InvoiceItem | Aggregate | **False** | The list of invoice items. | | InvoiceItem/ ItemSequenceNumber | Integer | **True** | A unique number that is related to this line item. | | InvoiceItem/ AmazonProductIdentifier | String | **False** | The Amazon Standard Identification Number (ASIN) of an item. | | InvoiceItem/ VendorProductIdentifier | String | **False** | The vendor-selected product identifier of the item. This identifier should be the same as the one that is provided in the purchase order. | | InvoiceItem/ PurchaseOrderNumber | String | **False** | The Amazon purchase-order number for this invoiced line item. The purchase-order number should be an eight-character alphanumeric code. This value is mandatory only when the invoice type is **Invoice**. | | InvoiceItem/ HsnCode | String | **False** | The HSN tax code. The HSN number cannot contain alphabetic characters. | | InvoiceItem/ InvoicedQuantityAmount | String | **True** | The quantity of an item. This value should not be zero. | | InvoiceItem/ InvoicedQuantityUnit | String | **False** | The unit of measure for the quantity. Allowed values are **Cases** and **Eaches**. | | InvoiceItem/ InvoicedQuantityUnitSize | Integer | **False** | The case size (if the unit-of-measure value is **Cases**). | | InvoiceItem/ NetCostAmount | String | **True** | The net cost amount as a decimal number with no loss of precision. | | InvoiceItem/ NetCostCurrencyCode | String | **True** | The three-digit currency code in ISO 4217 format. | | InvoiceItem/ CreditNote ReferenceInvoiceNumber | String | **False** | The original invoice number when you send a credit note that is related to an existing invoice. Only one invoice should be processed per credit note. This invoice number is mandatory for AP credit notes. | | InvoiceItem/ CreditNote DebitNoteNumber | String | **False** | The debit note number as generated by Amazon. This number is recommended for Returns and COOP credit notes. | | InvoiceItem/ CreditNote ReturnsReferenceNumber | String | **False** | A number that identifies the returns notice number. This number is mandatory for all Returns credit notes. | | InvoiceItem/ CreditNote GoodsReturnDate | String | **False** | A date and time that is defined according to ISO 8601. | | InvoiceItem/ CreditNote RMAId | String | **False** | The Returned Merchandise Authorization Id (if one is generated). | | InvoiceItem/ CreditNote CoopReference | String | **False** | The COOP reference number that is used for COOP agreement. Failure to provide the COOP reference number or the debit-note number might lead to a rejection of the credit note. | | InvoiceItem/ CreditNote Consignors ReferenceNumber | String | **False** | The consignor reference number (VRET number) if one is generated by Amazon. | | InvoiceItem/ TaxDetails | Aggregate | **False** | The individual tax details per line item. | | InvoiceItem/ ChargeDetails | String | **False** | The individual charge details per line item. | | InvoiceItem/ AllowanceDetails | Aggregate | **False** | The individual allowance details per line item. | # Amazon Marketplace Submit Vendor Shipment Confirmations Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-submit-vendor-shipment-confirmations Submit shipment confirmations for vendor orders in Amazon Marketplace. The Submit Vendor Shipment Confirmations action submits shipment confirmations for vendor orders. This action belongs at the end of a flow. User input for this action is provided solely through the input XML. ## XML Mapping You must connect an [XML Map](../xml-map/xml-map) connector to the Amazon Marketplace connector to map the required input parameters, as shown in the image below. ![Amazon Marketplace Submit Vendor Shipment Confirmations XML Mapping](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_marketplace_SVSC.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=47df8a6c30073851e206634b6aad8aed) ## Input Parameters The following table lists the input parameters that you can map with the XML Map connector interface. | Name | Data Type | Required? | Description | | - | :-: | :-: | - | | Shipment Identifier | String | **True** | The unique shipment ID. | | ShipmentConfirmation Type | String | **True** | The shipment confirmation type. This parameter indicates whether this shipment confirmation is the initial confirmation or whether it is intended to replace a shipment confirmation that is already posted. The allowed values are **Original** and **Replace**. | | ShipmentType | String | **False** | The type of shipment. Allowed values are **TruckLoad**, **LessThanTruckLoad**, and **SmallParcel**. | | ShipmentStructure | String | **False** | The shipment hierarchical structure. Allowed values are **PalletizedAssortmentCase**, **LooseAssortmentCase**, **PalletOfItems**, **PalletizedStandardCase**, **LooseStandardCase**, **MasterPallet**, and **MasterCase**. | | TransportationDetails CarrierScac | String | **False** | The code that identifies the carrier for the shipment. | | TransportationDetails CarrierShipment ReferenceNumber | String | **False** | A unique number that is assigned by the carrier. This field is also known as PRO number. | | TransportationDetails TransportationMode | String | **False** | The mode of transportation for this shipment. Allowed values are **Road**, **Air**, and **Ocean**. | | TransportationDetails BillOfLadingNumber | String | **False** | The Bill Of Lading (BOL) number. This number is the unique number that is assigned by the vendor. | | AmazonReferenceNumber | String | **False** | The Amazon Reference Number. This number is a unique identifier that is generated by Amazon for all Collect/WePay shipments. | | ShipmentConfirmation Date | Datetime | **True** | The date on which the shipment confirmation is submitted. | | ShippedDate | Datetime | **False** | The date and time of the departure of the shipment from the vendor's location. | | EstimatedDeliveryDate | Datetime | **False** | The date and time by which the shipment is expected to reach the buyer's warehouse. | | SellingPartyId | String | **True** | The assigned identification for the selling party. | | SellingPartyAddress | Aggregate | **False** | An aggregate representation of the selling party's address. | | SellingPartyAddress/ Name | String | **True** | The seller's name. | | SellingPartyAddress/ AddressLine1 | String | **True** | The first line of the selling party's address. | | SellingPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | SellingPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | SellingPartyAddress/ City | String | **False** | The city where the selling party is located. | | SellingPartyAddress/ County | String | **False** | The county where the selling party is located. | | SellingPartyAddress/ District | String | **False** | The district where the selling party is located. | | SellingPartyAddress/ StateOrRegion | String | **False** | The state or region where the selling party is located. | | SellingPartyAddress/ PostalCode | String | **False** | The postal code for the address. This code consists of a series of letters, digits, or both. | | SellingPartyAddress/ CountryCode | String | **False** | The two-digit country code in ISO 3166-1 alpha-2 format. | | SellingPartyAddress/ Phone | String | **False** | The phone number for the selling party that is located at that address. | | SellingParty TaxRegistrationType | String | **False** | The tax registration type for the entity. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | SellingParty TaxRegistrationNumber | String | **False** | The tax registration number for the entity (for example, the VAT Id). | | ShipFromPartyId | String | **True** | The assigned identification for the ship-from party. | | ShipFromPartyAddress | Aggregate | **False** | The identification of the shipper, by address. | | ShipFromPartyAddress/ Name | String | **True** | The shipper's name. | | ShipFromPartyAddress/ AddressLine1 | String | **True** | The first line of the shipper's address. | | ShipFromPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | ShipFromPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | ShipFromPartyAddress/ City | String | **False** | The city where the shipper is located. | | ShipFromPartyAddress/ County | String | **False** | The county where the shipper is located. | | ShipFromPartyAddress/ District | String | **False** | The district where the shipper is located. | | ShipFromPartyAddress/ StateOrRegion | String | **False** | The state or region where the shipper party is located. | | ShipFromPartyAddress/ PostalCode | String | **False** | The postal code for the shipper's address. This code consists of a series of letters, digits, or both. | | ShipFromPartyAddress/ CountryCode | String | **False** | The two-digit country code in ISO 3166-1 alpha-2 format. | | ShipFromPartyAddress/ Phone | String | **False** | The shipper's phone number. | | ShipFromPartyAddress TaxRegistrationType | String | **False** | The tax registration type for the entity. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | ShipFromPartyAddress TaxRegistrationNumber | String | **False** | The tax registration number for the entity (for example, the VAT Id). | | ShipToPartyId | String | **True** | The assigned identification for the ship-to party. | | ShipToPartyAddress | Aggregate | **False** | The identification of the ship-to party, by address. | | ShipToPartyAddress/ Name | String | **True** | The ship-to party's name. | | ShipToPartyAddress/ AddressLine1 | String | **True** | The first line of the ship-to party's address. | | ShipToPartyAddress/ AddressLine2 | String | **False** | Additional address information (if required). | | ShipToPartyAddress/ AddressLine3 | String | **False** | Additional address information (if required). | | ShipToPartyAddress/ City | String | **False** | The city where the ship-to party is located. | | ShipToPartyAddress/ County | String | **False** | The county where the ship-to party is located. | | ShipToPartyAddress/ District | String | **False** | The district where the ship-to party is located. | | ShipToPartyAddress/ StateOrRegion | String | **False** | The state or region where the ship-to party is located. | | ShipToPartyAddress/ PostalCode | String | **False** | The postal code for the ship-to address. This code consists of a series of letters, digits, or both. | | ShipToPartyAddress/ CountryCode | String | **False** | The two-digit country code in ISO 3166-1 alpha-2 format. | | ShipToPartyAddress/ Phone | String | **False** | The phone number for the ship-to party that is located at that address. | | ShipToParty TaxRegistrationType | String | **False** | The tax registration type for the ship-to party. Allowed values are **VAT** (the value-added tax) and **GST** (the goods-and-services tax). | | ShipTo TaxRegistrationNumber | String | **False** | The tax registration number for the ship-to party (for example, the VAT Id). | | ShipmentMeasurements | String | **False** | The JSON aggregate representation of the shipment measurement details. | | ImportDetails | String | **False** | The JSON aggregate representation of the import details. | | ShippedItems | Aggregate | **True** | An aggregate representation of the items in this shipment. | | ShippedItems/ ItemSequenceNumber | String | **True** | The line-item sequence number for the item. | | ShippedItems/ AmazonProductIdentifier | String | **False** | The Amazon Standard Identification Number (ASIN) of an item. | | ShippedItems/ VendorProductIdentifier | String | **False** | The vendor-selected product identification of the item. | | ShippedItems/ ShippedQuantityAmount | Integer | **True** | The shipped quantity. This value should not be zero. | | ShippedItems/ ShippedQuantityUnit | String | **True** | The unit of measure for the shipped quantity. | | ShippedItems/ ShippedQuantityUnitSize | Integer | **False** | The unit size for the shipped quantity. | | ShippedItems/ PurchaseOrderNumber | String | **False** | The Amazon purchase order (PO) number for the shipped item that is being confirmed. The PO number should be formatted as an eight-character, alphanumeric code. | | ShippedItems/ LotNumber | String | **False** | The lot number of the shipped quantity. | | ShippedItems/ ExpiryManufacturerDate | Datetime | **False** | The production, packaging, or assembly date that is determined by the manufacturer. This date's meaning is determined based on the trade-item context. | | ShippedItems/ ExpiryDate | Datetime | **False** | The date that determines the limit of consumption or use of a product. This date's meaning is determined based on the trade item context. | | ShippedItems/ ExpiryAfterDuration Unit | Datetime | **False** | The unit for the duration after the manufacturing date during which the product is valid for consumption. | | ShippedItems/ ExpiryAfterDuration Value | Integer | **False** | The value for the duration in terms of the duration unit. | | ShippedItems/ MaximumRetailPrice CurrencyCode | String | **False** | The three-digit currency code, in ISO 4217 format, for the maximum retail prices. | | ShippedItems/ MaximumRetailPrice Amount | String | **False** | The maximum retail price for the shipped items as a decimal number with no loss of precision. | | ShippedItems/ HandlingCode | String | **False** | The code that identifies the instructions about how the specified item, carton, or pallet should be handled. | | Cartons | Aggregate | **True** | An aggregate representation of the cartons in this shipment. | | Cartons/ CartonIdentifiers | Aggregate | **False** | A list of carton identifiers. | | Cartons/ CartonIdentifiers/ Container Identification Type | String | **True** | The container identification type. | | Cartons/ CartonIdentifiers/ Container Identification Number | String | **True** | The container identification number that adheres to the definition of the container identification type. | | Cartons/ CartonSequenceNumber | String | **True** | The sequence number for the carton. The first carton is numbered 001, the second 002, and so on. This number is used as a reference to refer to the carton from the pallet level. | | Cartons/ DimensionsLength | String | **False** | The length of the container. | | Cartons/ DimensionsWidth | String | **False** | The width of the container. | | Cartons/ DimensionsHeight | String | **False** | The height of the container. | | Cartons/ WeightValue | String | **False** | The weight measurement value for the container as a decimal number with no loss of precision. | | Cartons/ WeightUnit | String | **False** | The unit of measure for the weight of the container. | | Cartons/ TrackingNumber | String | **False** | The tracking number for the container. This number is required for every carton in the small-parcel shipments. | | Cartons/ CartonsItems | Aggregate | **False** | A list of carton item details. | | Cartons/ CartonsItems/ ItemReference | String | **True** | The reference number for the item. Be sure to provide the item sequence number from the **items** segment to refer to that item's details here. | | Cartons/ CartonsItems/ ShippedQuantityAmount | Integer | **True** | The number of units that are shipped for a specific item at a shipment level. If the item is present only in certain cartons within the shipment, provide this at the appropriate carton level. | | Cartons/ CartonsItems/ ShippedQuantityUnit | String | **True** | The unit of measure for the shipped quantity. | | Cartons/ CartonsItems/ ShippedQuantityUnitSize | Integer | **False** | The case size (if product is ordered by the case). Otherwise, the value is **1**. | | Cartons/ CartonsItems/ PurchaseOrderNumber | String | **False** | The Amazon purchase order (PO) number for the shipment that is being confirmed. If the items in this shipment belong to multiple PO numbers that are in particular carton within the shipment, then provide the PO number at the carton level. The PO numbers should be formatted as an eight-character, alphanumeric code. | | Cartons/ CartonsItems/ LotNumber | String | **False** | The batch or lot number for the cartons. This number associates cartons with information that the manufacturer considers relevant for traceability of the cartons to which the element string is applied. The data might refer to the itself or to items within the carton. This field is mandatory for all perishable items. | | Cartons/ CartonsItems/ ExpiryManufacturerDate | Datetime | **False** | The production, packaging, or assembly date that is determined by the manufacturer. This date's meaning is determined based on the trade-item context. | | Cartons/ CartonsItems/ ExpiryDate | Datetime | **False** | The date that determines the limit of consumption or use of a product. This date's meaning is determined based on the trade item context. | | Cartons/ CartonsItems/ ExpiryAfterDuration Unit | String | **False** | The unit for the duration after the manufacturing date during which the product is valid for consumption. | | Cartons/ CartonsItems/ ExpiryAfterDuration Value | Integer | **False** | The value for the duration in terms of the duration unit. | | Cartons/ CartonsItems/ MaximumRetailPrice CurrencyCode | String | **False** | The three-digit currency code, in ISO 4217 format, for the maximum retail prices. | | Cartons/ CartonsItems/ MaximumRetailPrice Amount | String | **False** | The maximum retail price for the shipped items as a decimal number with no loss of precision. | | Cartons/ CartonsItems/ HandlingCode | String | **False** | A code that identifies the instructions about how the specified carton should be handled. | | Pallets | Aggregate | **False** | An aggregate representation of the pallets in this shipment. | | Pallets/ PalletIdentifiers | Aggregate | **True** | A list of pallet identifiers. | | Pallets/ PalletIdentifiers/ Container Identication Type | String | **True** | The container identification type. Allowed values are **SSCC**, **AMZNCC**, **GTIN**, **BPS**, and **CID**. | | Pallets/ PalletIdentifiers/ Container Identification Number | String | **True** | The container identification number that adheres to the definition of the container identification type. | | Pallets/ Tier | Integer | **False** | The number of layers per pallet. | | Pallets/ Block | Integer | **False** | The number of cartons per layer on the pallet. | | Pallets/ DimensionsLength | String | **False** | The length of the pallet. | | Pallets/ DimensionsWidth | String | **False** | The width of the pallet. | | Pallets/ DimensionsHeight | String | **False** | The height of the pallet. | | Pallets/ DimensionsUnit | String | **False** | The unit of measure for the dimensions of the pallet. | | Pallets/ WeightValue | String | **False** | The weight measurement value for pallet as a decimal number with no loss of precision. | | Pallets/ WeightUnit | String | **False** | The unit of measure for the weight of the pallet. | | Pallets/ CartonsCount | Integer | **False** | The number of cartons that are present in the shipment. Provide the carton count only for unpalletized shipments. | | Pallets/ CartonReferenceNumbers | String | **False** | An array of reference numbers for the cartons that are part of this pallet and shipment. Provide the carton sequence number from the **cartons** segment to refer to that carton's details here. | | Pallets/ PalletsItems | Aggregate | **False** | A list of pallet item details. | | Pallets/ PalletsItems/ ItemReference | String | **True** | The reference number for the item. Provide the item sequence number from the **items** segment to refer to that item's details here. | | Pallets/ PalletsItems/ ShippedQuantityAmount | Integer | **True** | The number of units that are shipped for a specific item at a shipment level. If the item is present only in certain cartons or pallets within the shipment, provide this at the appropriate carton or pallet level. | | Pallets/ PalletsItems/ ShippedQuantityUnit | String | **True** | The unit of measure for the shipped quantity. | | Pallets/ PalletsItems/ ShippedQuantityUnitSize | Integer | **False** | The case size (if product is ordered by the case). Otherwise, the value is **1**. | | Pallets/ PalletsItems/ PurchaseOrderNumber | String | **False** | The Amazon purchase order (PO) number for the shipment that is being confirmed. If the items in this shipment belong to multiple PO numbers that are in particular pallet within the shipment, then provide the PO number at the pallet level. The PO numbers should be formatted as an eight-character, alphanumeric code. | | Pallets/ PalletsItems/ LotNumber | String | **False** | The batch or lot number for the pallet. This number associates a pallet with information that the manufacturer considers relevant for traceability of the pallet to which the element string is applied. The data might refer to the pallet itself or to items within the pallet. This field is mandatory for all perishable items. | | Pallets/ PalletsItems/ ExpiryManufacturerDate | Datetime | **False** | The production, packaging, or assembly date that is determined by the manufacturer. This date's meaning is determined based on the trade-item context. | | Pallets/ PalletsItems/ ExpiryDate | Datetime | **False** | The date that determines the limit of consumption or use of a product. This date's meaning is determined based on the trade item context. | | Pallets/ PalletsItems/ ExpiryAfterDuration Unit | String | **False** | The unit for the duration after the manufacturing date during which the product is valid for consumption. | | Pallets/ PalletsItems/ ExpiryAfterDuration Value | Integer | **False** | The value for the duration in terms of the duration unit. | | Pallets/ PalletsItems/ MaximumRetailPrice CurrencyCode | String | **False** | The three-digit currency code, in ISO 4217 format, for the maximum retail prices. | | Pallets/ PalletsItems/ MaximumRetailPrice Amount | String | **False** | The maximum retail price for the shipped items as a decimal number with no loss of precision. | | Pallets/ PalletsItems/ HandlingCode | String | **False** | A code that identifies the instructions about how the specified pallet should be handled. | # Amazon Marketplace Update Inventory Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-update-inventory Update inventory quantities for specified items in Amazon Marketplace. The Update Inventory action updates your inventory quantities (stock levels) in Amazon Marketplace. This action belongs at the end of a flow. ## XML Mapping You must connect an [XML Map](../xml-map/xml-map) connector to the Amazon Marketplace connector and map the inventory update values. In the example below, the order information appears in the **Source** column, and the Amazon Marketplace inventory parameters appear in the **Destination** column. ![Amazon Marketplace Update Inventory XML Mapping](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_updateinventory_mapping.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=db772a17325c25b7b260756e2c7e7902) ## Connector Configuration In the configuration pane for the Amazon Marketplace connector, enter a comma-separated list of **Marketplace Ids** for the items that you want to update. ![Amazon Marketplace Update Inventory Marketplace Ids](https://mintcdn.com/cdata-arc/pXe_oAwLTI7MI6MK/26.4/images/amazon_updateinventory_ids.png?fit=max\&auto=format\&n=pXe_oAwLTI7MI6MK\&q=85\&s=79e680762c24fbe5e7c00d65785b049a) # Amazon Marketplace Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-upsert Insert or update Amazon Marketplace data from an Arc flow. The Upsert action updates or inserts Amazon Marketplace data. By default, if a record already exists in Amazon Marketplace, an update is performed on the existing data in Amazon Marketplace using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Amazon Marketplace Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Amazon Marketplace. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/amazon-marketplace/amazon-marketplace-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Amazon Marketplace connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Amazon Marketplace Upsert connector at the end: ![](/26.4/images/amazon_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Amazon Marketplace\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Amazon Marketplace.
# AMQP Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/amqp Connect Arc to an AMQP broker to produce and consume messages using the Advanced Message Queuing Protocol. The Advanced Message Queueing Protocol (AMQP) connector allows you to integrate AMQP into your data flow by performing Produce and Consume actions on your data. ## Key Capabilities * Bidirectional message queueing with produce and consume actions * Support for multiple authentication schemes (None, SASL Anonymous, SASL Plain) * TLS encryption support for secure connections * Flexible message value types including JSON composite data support Follow the steps below to connect CData Arc to AMQP. ## Establish a Connection You must first establish a connection to AMQP through one of these two methods: * Add a AMQP connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the CData Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **AMQP**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to AMQP can be re-used across multiple AMQP connectors. After opening the connection dialogue, follow these steps: 1. Enter the connection information: * **Name** — the static name of the connection. * **Type** — this is always set to AMQP. * **Remote Host** — the hostname or IP address of your AMQP remote host. * **Remote Port** — the port for your remote host connection. * **Use TLS** — whether to use TLS to encrypt the connection. Set this according to your AMQP configuration. * **Container Id** — (optional) the ID value for the container in your AMQP connection that you want to access. * **Auth Scheme** — the authentication scheme to use for the connection. Options are **None**, **SASL Anonymous**, and **SASL Plain**. If you select **SASL Anonymous** or **SASL Plain** authentication, enter the following information: * **User** — the username to use when connecting to AMQP. * **Password** — the password for the user. 2. Click **Test Connection** to ensure that Arc can connect to AMQP with the provided information. If an error occurs, check all fields and try again. 3. Click **Add Connection** to finalize the connection. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Connection** | Select the connection you created in the previous steps. | | **Target** | (Optional) For the [Produce](#produce) action, supply the address of the node that should act as the receiver. For the [Consume](#consume) action, supply the address of the node that the connector should receive data from. | | **Action** | The action that the connector should perform. See [Select an Action](#select-an-action) for more information. | | **Server Status** | (Only applicable when **Action** is **Consume**) The state of the connector's server: **Active** (the server is running; click **Stop Server** to stop it), **Inactive** (the server is not running; click **Start Server** to start it). To review the logs the server produces while it runs, see the [Logs tab](#logs-tab). | #### AMQP Message | Setting | Description | | - | - | | **Value Type** | The default value type for messages produced by the connector. Including a message header called *AMQP-ValueType* on input messages overrides this selection. | ##### JSON Composite Data Value Type One **Value Type** option is **JSON (Composite Data)**. The composite data JSON schema consists of `type-value` objects, which hold an AMQP typename and a value. The syntax for a `type-value` object is: ```json theme={null} { "type": "typename", "value": value } ``` The following table provides a complete list of valid typenames, and describes how the associated values should be represented: | Typename | Description | Value Format | | - | - | - | | null | Null | null | | boolean | Boolean | true or false | | ubyte | Unsigned byte | 0 to 255 | | ushort | Unsigned short | 0 to 65535 | | uint | Unsigned integer | 0 to 4294967295 | | ulong | Unsigned long | 0 to 18446744073709551615 | | byte | Byte | -128 to 127 | | short | Short | -32768 to 32767 | | int | Integer | -2147483648 to 2147483647 | | long | Long | -9223372036854775808 to 9223372036854775807 | | float | Float | \[IEEE 754 32-bit floating point number] | | double | Double | \[IEEE 754 64-bit floating point number] | | decimal | Decimal | Hex-encoded byte string like "0A1B2C3D" | | char | Char | "c" | | timestamp | Timestamp | \[Number of milliseconds since the Unix epoch (January 1, 1970 00:00:00 UTC)] | | uuid | UUID | Hex-encoded UUID string like "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" (hyphens optional, case-insensitive) | | binary | Binary data | Hex-encoded byte string like "0A1B2C3D" | | string | String | "UTF-8 string (correctly escaped for JSON)" | | symbol | Symbolic value | "ASCII string" | | array | AMQP array | JSON array with zero or more type-value objects of the same type | | list | AMQP list | JSON array with zero or more type-value objects | | map | AMQP map | JSON array with even number of type-value objects representing keys and values; no duplicate keys allowed | > [!NOTE] > > Typenames must be lowercase in order to be parsed correctly. ### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
#### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Logs Tab The **Logs** tab lists the service logs generated by this connector. Each row represents a log file and shows its date modified, file name, size, and status. Use the search bar or the status filter to find a specific log, click a file name to preview or download it, or click **Download All Logs** to download all of them as a single zip file. ## Select an Action After establishing a connection to AMQP, you must choose the action that the AMQP connector will perform. The table below outlines each action and where it belongs in a CData Arc flow: | Action | Description | Position in Flow | | - | - | - | | [Produce](#produce) | Accepts input data from a file or another connector and sends it to your AMQP connection. | End | | [Consume](#consume) | Listens for messages coming into the queue and sends incoming data down the flow through the Output path. | Beginning | ### Produce The **Produce** action sends input data to your AMQP connection. This data can come from other connectors or from files that you manually upload to the **Transactions** tab of the AMQP connector. In some AMQP configurations, you can set the value of the **Target** field to the address of the node that should act as the receiver. This might not be necessary in all configurations. ### Consume The **Consume** action checks for messages in the AMQP queue. Data processed through the **Consume** connector goes to the **Transactions** tab and travels down to the next steps of the Arc flow. In some AMQP configurations, you can set the value of the **Target** field to the address of the node that the connector should receive data from. This might not be necessary in all configurations. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Apache Kafka Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/apachekafka Connect Arc to Apache Kafka for data integration and synchronization. The Apache Kafka connector allows you to integrate Apache Kafka into your data flow by pushing or pulling data from Apache Kafka. ## Key Capabilities * Real-time message streaming with produce and consume capabilities * Multiple authentication schemes including SASL and Kerberos * Topic-based message organization with consumer group management * Configurable read duration to control how long to wait for messages to arrive Follow the steps below to connect Arc to Apache Kafka. ## Prerequisites To establish a connection with Apache Kafka, at a minimum users must have: * An Apache Kafka environment that consists of a valid Apache Kafka host server and server manager (the broker, which is commonly Zookeeper or KRaft). * Network connectivity from the Apache Kafka host to all Apache Kafka servers and server managers in the cluster containing the targeted topics. * Topics created in Apache Kafka to send and receive messages from the server. * A clear network path from Arc to the Apache Kafka server. > [!NOTE] > > CData does not provide setup assistance or support for the creation or configuration of the Apache Kafka server or server manager. For information on the setup of a Apache Kafka environment, consult the [Apache Kafka](https://kafka.apache.org/41/) documentation. ## Establish a Connection To allow Arc to use data from Apache Kafka, you must first establish a connection to Apache Kafka. There are two ways to establish this connection: * Add a Apache Kafka connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Apache Kafka**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Apache Kafka can be reused across multiple Apache Kafka connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Apache Kafka. * **Auth Scheme** — The authorization scheme to use for the connection. Options are **Auto**, **None**, **Plain**, **Scram**, and **Kerberos**. * **User** — (All schemes except **None**) The Apache Kafka username to use for logging in. * **Password** — (All schemes except **None**) The password for the user entered above. * **Bootstrap Servers** — The host/port pair to use for establishing the initial connection to Apache Kafka. If you are connecting to Confluent Cloud, you can find this on the Cluster settings. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Apache Kafka with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. In the **Topic** field, enter the Apache Kafka topic that you want to target. 7. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Apache Kafka, you must choose the action that the Apache Kafka connector should perform. The table below outlines each action and where it belongs in an Arc flow: | Action | Description | Position in Flow | | - | - | - | | [Produce](#produce) | Accepts input data from a file or another connector and sends it to Apache Kafka. | End | | [Consume](#consume) | Checks the queue for messages and sends any data that it gets down the flow through the Output path. | Middle | ### Produce The **Produce** action sends input data to Apache Kafka. This data can come from other connectors or from files that you manually upload to the Input tab of the Apache Kafka connector. The Apache Kafka connector sends the input data to the topic you entered in the **Topic** field of the **Configuration** section. #### Advanced Message Key and Partition Configuration The Apache Kafka connector supports specifying message keys and partitions when producing messages to Kafka topics, providing granular control over message distribution and ordering in multi-partition environments. ##### Message Key Configuration You can specify the Kafka message key using the Arc message header `MessageKey` or by using the **Key Header** field in the [Advanced Settings](#advanced-settings) section of the **Advanced** tab to tell Arc which header on the input message to use to determine the message key. This key determines which partition the message is sent to and enables message ordering guarantees within partitions. ##### Partition Configuration You can control which partition receives your message using these optional settings: * **Partition Header:** Defines the message header that the connector uses to set the partition. The default header name is `MessagePartition` but you can override it with any message header that contains the desired partition. * **Partition:** Provide the partition number in the [Advanced Settings](#advanced-settings) section on the **Advanced** tab. > [!NOTE] > > * If a partition header and specific partition number are both present, the message header takes precedence. > * If neither a message key nor a partition is specified, Kafka distributes messages across all partitions in the target topic. ### Consume The **Consume** action checks for messages in the Apache Kafka queue for the topic you entered in the **Topic** field of the **Configuration** section. You must set the following fields for this action: * **Consumer Group ID**: Specifies which group the consumers created by the connector should belong to. * **Read Duration**: The length of time (in seconds) that the connector waits for messages to arrive. The connector waits the full duration, regardless of the number of messages received. Data processed through the **Consume** connector goes to the Output tab and travels down to the next steps of the Arc flow. ## Additional Connection Configuration ### Advanced Tab #### Advanced Settings | Setting | Description | | - | - | | **Partition** | The partition number to send messages to. If this and **Partition Header** are supplied, the latter takes precedence. See [Partition Configuration](#partition-configuration) for details on this and the following setting. | | **Partition Header** | The name of the Arc message header used to specify the target partition. The default is `MessagePartition`. | | **Key Header** | The name of the Arc message header used to specify the message key. The default is `MessageKey`. See [Message Key Configuration](#message-key-configuration) for details. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Other Driver Settings** | Hidden properties that are used only in specific use cases. | #### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](./file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | #### Logging | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | #### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](./Performance-Settings.html) portion of the [Advanced Settings](./Advanced-Settings.html) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](./Performance-Settings.html) portion of the [Advanced Settings](./Advanced-Settings.html) page. | ### Alerts Tab *Settings related to configuring alerts and Service Level Agreements (SLAs).* #### Connector Email Settings Before you can execute SLAs, you need to set up email alerts for notifications. Clicking **Configure Alerts** opens a new browser window to the [Settings page](./Settings-Page.html) where you can set up system-wide alerts. See [Alerts](./Settings-Page.html#alerts) for more information. #### Service Level Agreement (SLA) Settings SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, click **Add Expected Volume Criteria**. * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * Set **Expect at least** to the minimum number of transactions (the volume) you expect to be processed, then use the **Every** fields to specify the time frame. * By default, the SLA is in effect every day. To change that, uncheck **Everyday** then check the boxes for the days of the week you want. * Use **And set status to 'At Risk'** to indicate when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](./Date-Formatters.html#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](./Global-Settings-Vault.html). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](./Date-Formatters.html#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # BigCommerce Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce Connect Arc to BigCommerce for data integration and synchronization The BigCommerce connector allows you to integrate BigCommerce into your data flow by pushing or pulling data from BigCommerce. ## Key Capabilities * E-commerce platform integration with store-specific access controls * Access token authentication for secure API connectivity * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to BigCommerce. ## Establish a Connection To allow Arc to use data from BigCommerce, you must first establish a connection to BigCommerce. There are two ways to establish this connection: * Add a BigCommerce connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **BigCommerce**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to BigCommerce can be reused across multiple BigCommerce connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to BigCommerce. * **Store ID** — The store hash for the BigCommerce account. * **Access Token** — The access token for establishing the connection. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to BigCommerce with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to BigCommerce, you must choose the action that the BigCommerce connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./bigcommerce-upsert) | Inserts or updates BigCommerce data. By default, if a record already exists in BigCommerce, an update is performed on the existing data in BigCommerce using the values provided from the input. | End | | [Lookup](./bigcommerce-lookup) | Retrieves values from BigCommerce and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from BigCommerce. It should be formatted as a SQL query against the BigCommerce tables. | Middle | | [Lookup Stored Procedure](./bigcommerce-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./bigcommerce-select) | Retrieves data from BigCommerce and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./bigcommerce-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./bigcommerce-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # BigCommerce Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-ac Advanced configuration options for the BigCommerce connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the BigCommerce connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The BigCommerce connector models BigCommerce tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and BigCommerce values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The BigCommerce connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into BigCommerce. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The BigCommerce connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the BigCommerce connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by BigCommerce. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by BigCommerce, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from BigCommerce. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure BigCommerce connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # BigCommerce Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the BigCommerce connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the BigCommerce connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # BigCommerce Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-lookup Retrieve values from BigCommerce and insert them into an Arc message in the flow. The Lookup action retrieves values from BigCommerce and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the BigCommerce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the BigCommerce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the BigCommerce connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the BigCommerce connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from BigCommerce tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The BigCommerce connector can perform Lookup queries against a local cache instead of querying BigCommerce directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query BigCommerce on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries BigCommerce directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to BigCommerce.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # BigCommerce Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-lsp Use a stored procedure to retrieve values from BigCommerce and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the BigCommerce connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the BigCommerce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the BigCommerce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# BigCommerce Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-select Retrieve data from BigCommerce and bring it into Arc. The Select action retrieves data from BigCommerce and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries BigCommerce. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to BigCommerce, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from BigCommerce. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in BigCommerce relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what BigCommerce supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured BigCommerce connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a BigCommerce Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **BigCommerce\_Select** connector pulls data from a BigCommerce connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **BigCommerce\_Select** then passes down the flow from **CSV1**.
# BigCommerce Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-upsert Insert or update BigCommerce data from an Arc message in the flow. The Upsert action updates or inserts BigCommerce data. By default, if a record already exists in BigCommerce, an update is performed on the existing data in BigCommerce using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The BigCommerce Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from BigCommerce. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/bigcommerce/bigcommerce-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured BigCommerce connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a BigCommerce Upsert connector at the end: ![](/26.4/images/bigcommerce_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **BigCommerce\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to BigCommerce.
# Database Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-ac Reference for advanced settings available on the Database connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the Database connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Database connector models Database tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Database values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Database connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Database. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Database connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Database connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Database. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Database, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Database. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Database connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Database Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-esp Treat data coming into the connector as input for a stored procedure, then pass the result down the flow. The Execute Stored Procedures action treats data coming into the Database connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Database connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Database Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/database/database-select Retrieve data from the database and bring it into Arc. The Select action retrieves data from Database and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Database. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Database, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Database. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/database/database-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Database relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Database supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Database connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Database Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/database_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **Database\_Select** connector pulls data from a Database connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Database\_Select** then passes down the flow from **CSV1**.
# Databricks Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks Connect Arc to Databricks for data integration and synchronization The Databricks connector allows you to integrate Databricks into your data flow by pushing or pulling data from Databricks. ## Key Capabilities * Databricks lakehouse platform integration with Personal Access Token and Azure Service Principal authentication * HTTP Path configuration for targeting specific Databricks compute resources and clusters * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Databricks. ## Establish a Connection To allow Arc to use data from Databricks, you must first establish a connection to Databricks. There are two ways to establish this connection: * Add a Databricks connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Databricks**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Databricks can be re-used across multiple Databricks connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter the connection information: * **Name** — The static name of the connection. * **Type** — This is always set to Databricks. * **Auth Scheme** — The authentication scheme to use for the connection. Options are **Personal Access Token** and **Azure Service Principal**. 2. Create the connection according to your selected Auth scheme: * **Personal Access Token** — Enter the following information: * **Server** — The host name or IP address of the server hosting Databricks. * **HTTP Path** — The endpoint path on the Databricks instance where API requests should be sent, typically in the format `/sql/1.0/warehouses/{warehouse_id}`. * **Access Token** — Your Databricks access token. * **Azure Service Principal** — Enter the following information: * **Server** — The host name or IP address of the server hosting Databricks. * **HTTP Path** — The endpoint path on the Databricks instance where API requests should be sent, typically in the format `/sql/1.0/warehouses/{warehouse_id}`. * **AzureTenantId** — The tenant Id of your Microsoft Azure Active Directory. * **AzureClientId** — The application (client) Id of your Microsoft Azure Active Directory application. * **Azure Client Secret** — The application (client) secret of your Microsoft Azure Active Directory application. * **Azure Subscription Id** — The subscription Id of your Microsoft Azure Databricks Service Workspace. * **Azure Resource Group** — The resource group name of your Microsoft Azure Databricks Service workspace. * **Azure Workspace** — The name of your Microsoft Azure Databricks Service Workspace. 3. If necessary, click **Advanced** to open the drop-down menu of advanced connection settings. These settings should not be needed in most cases. 4. Click **Test Connection** to ensure that Arc can connect to Databricks with the provided information. If an error occurs, check all fields and try again. 5. Click **Add Connection** to finalize the connection. 6. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 7. Click **Save Changes**. ## Select an Action After establishing a connection to Databricks, you must choose the action that the Databricks connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./databricks-upsert) | Inserts or updates Databricks data. By default, if a record already exists in Databricks, an update is performed on the existing data in Databricks using the values provided from the input. | End | | [Lookup](./databricks-lookup) | Retrieves values from Databricks and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Databricks. It should be formatted as a SQL query against the Databricks tables. | Middle | | [Lookup Stored Procedure](./databricks-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./databricks-select) | Retrieves data from Databricks and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./databricks-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./databricks-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Databricks Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-ac Advanced configuration options for the Databricks connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Databricks connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Databricks connector models Databricks tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Databricks values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Databricks connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Databricks. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Databricks connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Databricks connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Databricks. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Databricks, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Databricks. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Databricks connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Databricks Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Databricks connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Databricks connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Databricks Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-lookup Retrieve values from Databricks and insert them into an Arc message in the flow. The Lookup action retrieves values from Databricks and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Databricks connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Databricks connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Databricks connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Databricks connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Databricks tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Databricks connector can perform Lookup queries against a local cache instead of querying Databricks directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Databricks on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Databricks directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Databricks.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Databricks Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-lsp Use a stored procedure to retrieve values from Databricks and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Databricks connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Databricks connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Databricks connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Databricks Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-select Retrieve data from Databricks and bring it into Arc. The Select action retrieves data from Databricks and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Databricks. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Databricks, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Databricks. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/databricks/databricks-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Databricks relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Databricks supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Databricks connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Databricks Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Databricks\_Select** connector pulls data from a Databricks connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Databricks\_Select** then passes down the flow from **CSV1**.
# Databricks Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/databricks/databricks-upsert Insert or update Databricks data from an Arc message in the flow. The Upsert action updates or inserts Databricks data. By default, if a record already exists in Databricks, an update is performed on the existing data in Databricks using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Databricks Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Databricks. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/databricks/databricks-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Databricks connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Databricks Upsert connector at the end: ![](/26.4/images/databricks_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Databricks\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Databricks.
# DB2 Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2 Connect Arc to DB2 for data integration and synchronization The DB2 connector allows you to integrate DB2 into your data flow by pushing or pulling data from DB2. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to DB2. ## Prerequisites Before using a DB2 connector, you must already have a live DB2 database server and a clear network path to that server from Arc. If you are using the Java edition, all dependencies are shipped with the connector. However, if you are using the .NET edition you must also install and configure the dependencies that DB2 requires by following these steps: 1. Download the *IBM Data Server Driver Package (DS Driver)* from [https://www.ibm.com/support/pages/download-initial-version-115-clients-and-drivers](https://www.ibm.com/support/pages/download-initial-version-115-clients-and-drivers). 2. Run setup.exe as an administrator. CData Arc recommends that you use the default values in the installer unless you have a specific reason not to. 3. Verify that the installation was successful by checking that: * The IBM.Data.DB2.dll and related dlls are present in `C:\Program Files\IBM\IBM DATA SERVER DRIVER\bin\netf40` * The installation modified the PATH environment variable of the machine. You should see these paths (or a variation thereof): * `C:\Program Files\IBM\IBM DATA SERVER DRIVER\bin\netf40` * `C:\PROGRA~1\IBM\IBMDAT~1\BIN` 4. Restart Arc so that it can see the new driver. ## Establish a Connection To allow Arc to use data from DB2, you must first establish a connection to DB2. There are two ways to establish this connection: * Add a DB2 connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **DB2**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to DB2 can be reused across multiple DB2 connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to DB2. * **Server** — The address of the DB2 server you want to connect to. * **Port** — The port to use when connecting to the server. * **Database** — The name of the database you want to target. * **User** — The username to use for logging in. * **Password** — The password for the user entered above. * **Other** — Other authentication information if needed. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to DB2 with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to DB2, you must choose the action that the DB2 connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./db2-upsert) | Inserts or updates DB2 data. By default, if a record already exists in DB2, an update is performed on the existing data in DB2 using the values provided from the input. | End | | [Lookup](./db2-lookup) | Retrieves values from DB2 and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from DB2. It should be formatted as a SQL query against the DB2 tables. | Middle | | [Lookup Stored Procedure](./db2-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./db2-select) | Retrieves data from DB2 and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./db2-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./db2-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # DB2 Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-ac Reference for advanced settings available on the DB2 connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the DB2 connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The DB2 connector models DB2 tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and DB2 values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The DB2 connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into DB2. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The DB2 connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the DB2 connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by DB2. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by DB2, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from DB2. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure DB2 connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # DB2 Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-esp Use a DB2 stored procedure to process data coming into the connector and pass the result down the flow. The Execute Stored Procedures action treats data coming into the DB2 connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the DB2 connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # DB2 Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-lookup Retrieve values from DB2 and insert them into an Arc message in the flow. The Lookup action retrieves values from DB2 and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the DB2 connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the DB2 connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the DB2 connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the DB2 connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from DB2 tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The DB2 connector can perform Lookup queries against a local cache instead of querying DB2 directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query DB2 on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries DB2 directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to DB2.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # DB2 Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-lsp Use a DB2 stored procedure to look up values and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the DB2 connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the DB2 connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the DB2 connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# DB2 Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-select Retrieve data from DB2 and bring it into Arc. The Select action retrieves data from DB2 and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries DB2. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to DB2, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from DB2. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/db2/db2-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in DB2 relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what DB2 supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured DB2 connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a DB2 Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/db2_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **DB2\_Select** connector pulls data from a DB2 connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **DB2\_Select** then passes down the flow from **CSV1**.
# DB2 Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2/db2-upsert Insert or update data in DB2 from an Arc message in the flow. The Upsert action updates or inserts DB2 data. By default, if a record already exists in DB2, an update is performed on the existing data in DB2 using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The DB2 Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from DB2. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/db2/db2-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured DB2 connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a DB2 Upsert connector at the end: ![](/26.4/images/db2_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **DB2\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to DB2.
# DB2 for IBM i Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi Connect Arc to DB2 for IBM i for data integration and synchronization The DB2 for IBM i connector allows you to integrate DB2 for IBM i into your data flow by pushing or pulling data from DB2 for IBM i in the form of XML. ## Key Capabilities * IBM DB2 for IBM i (AS/400) integration optimized for legacy system connectivity and native data operations * XML-based data exchange for seamless integration between legacy and modern systems * Standard database connector actions with DB2 for IBM i-specific optimizations Follow the steps below to connect Arc to DB2 for IBM i. ## Prerequisites Before using a DB2 for IBM i connector, you must already have a live DB2 for IBM i database server and a clear network path to that server from Arc. Most of the required dependencies are shipped with the connector. However, if you are using the .NET edition, you must install IBM.Data.DB2.iSeries.dll. 1. Download the *ACS Windows App Pkg English (64bit)* from [https://www.ibm.com/resources/mrs/assets/DownloadList?source=swg-ia\&lang=en\_US](https://www.ibm.com/resources/mrs/assets/DownloadList?source=swg-ia\&lang=en_US). 2. Run setup.exe as an administrator. CData Arc recommends that you use the default values in the installer unless you have a specific reason not to. 3. Verify that the installation was successful by checking that the IBM.Data.DB2.iSeries.dll is present in `C:\Program Files (x86)\IBM\Client Access\`. 4. Restart Arc so that it can see the new driver. ## Establish a Connection To allow Arc to use data from DB2 for IBM i, you must first establish a connection to DB2 for IBM i. There are two ways to establish this connection: * Add a DB2 for IBM i connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **DB2 for IBM i**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to DB2 for IBM i can be reused across multiple DB2 for IBM i connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to DB2 for IBM i. * **Server** — The address of the DB2 IBM i server you want to connect to. * **User** — The username to use for logging in. * **Password** — The password for the user entered above. * **Other** — Other authentication information if needed. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to DB2 for IBM i with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to DB2 for IBM i, you must choose the action that the DB2 for IBM i connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./db2ibmi-upsert) | Inserts or updates DB2 for IBM i data. By default, if a record already exists in DB2 for IBM i, an update is performed on the existing data in DB2 for IBM i using the values provided from the input. | End | | [Lookup](./db2ibmi-lookup) | Retrieves values from DB2 for IBM i and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from DB2 for IBM i. It should be formatted as a SQL query against the DB2 for IBM i tables. | Middle | | [Lookup Stored Procedure](./db2ibmi-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./db2ibmi-select) | Retrieves data from DB2 for IBM i and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./db2ibmi-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./db2ibmi-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # DB2 for IBM i Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-ac Reference for advanced settings available on the DB2 for IBM i connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the DB2 for IBM i connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The DB2 for IBM i connector models DB2 for IBM i tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and DB2 for IBM i values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The DB2 for IBM i connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into DB2 for IBM i. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The DB2 for IBM i connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the DB2 for IBM i connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by DB2 for IBM i. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by DB2 for IBM i, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from DB2 for IBM i. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure DB2 for IBM i connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # DB2 for IBM i Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-esp Treat data coming into the connector as input for a stored procedure, then pass the result down the flow. The Execute Stored Procedures action treats data coming into the DB2 for IBM i connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the DB2 for IBM i connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # DB2 for IBM i Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-lookup Retrieve values from DB2 for IBM i and insert them into an Arc message in the flow. The Lookup action retrieves values from DB2 for IBM i and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the DB2 for IBM i connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the DB2 for IBM i connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the DB2 for IBM i connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the DB2 for IBM i connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from DB2 for IBM i tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The DB2 for IBM i connector can perform Lookup queries against a local cache instead of querying DB2 for IBM i directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query DB2 for IBM i on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries DB2 for IBM i directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to DB2 for IBM i.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # DB2 for IBM i Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-lsp Treat data coming into the connector as input for a stored procedure, then insert the result into an existing Arc message in the flow. The Lookup Stored Procedure action treats data coming into the DB2 for IBM i connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the DB2 for IBM i connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the DB2 for IBM i connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# DB2 for IBM i Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-select Retrieve data from DB2 for IBM i and bring it into Arc. The Select action retrieves data from DB2 for IBM i and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries DB2 for IBM i. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to DB2 for IBM i, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from DB2 for IBM i. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in DB2 for IBM i relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what DB2 for IBM i supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured DB2 for IBM i connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a DB2 for IBM i Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/db2_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **DB2 for IBM i\_Select** connector pulls data from a DB2 for IBM i connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **DB2 for IBM i\_Select** then passes down the flow from **CSV1**.
# DB2 for IBM i Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-upsert Insert or update DB2 for IBM i data from an Arc message in the flow. The Upsert action updates or inserts DB2 for IBM i data. By default, if a record already exists in DB2 for IBM i, an update is performed on the existing data in DB2 for IBM i using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The DB2 for IBM i Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from DB2 for IBM i. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/db2ibmi/db2ibmi-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured DB2 for IBM i connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a DB2 for IBM i Upsert connector at the end: ![](/26.4/images/db2ibmi_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **DB2 for IBM i\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to DB2 for IBM i.
# Dynamics 365 Business Central Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc Connect Arc to Dynamics 365 Business Central for data integration and synchronization The Dynamics 365 Business Central connector allows you to integrate Dynamics 365 Business Central into your data flow by pushing or pulling data from Dynamics 365 Business Central. ## Key Capabilities * Microsoft Dynamics 365 Business Central integration with multiple Azure authentication schemes * Organization URL and company-specific configuration for multi-tenant business environments * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Dynamics 365 Business Central. ## Establish a Connection To allow Arc to use data from Dynamics 365 Business Central, you must first establish a connection to Dynamics 365 Business Central. There are two ways to establish this connection: * Add a Dynamics 365 Business Central connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Dynamics 365 Business Central**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Dynamics 365 Business Central can be reused across multiple Dynamics 365 Business Central connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter a name for the connection. 2. The default **Organization URL** is correct for most users. However, if the default value does not match your configuration, enter the correct Organization URL. 3. Enter your **Company**. 4. Select your **Auth Scheme** from the drop-down menu. Options are **AzureAD**, **AzureServicePrincipal**, **AzureMSI**, and **AccessKey**. 5. If your auth scheme is AzureServicePrincipal, provide the **Azure Tenant Id**, **Client Id**, and **Client Secret**. If your auth scheme is AccessKey, provide the **User** and **Access Key**. 6. Click **Connect**. 7. A new window opens for logging into Dynamics 365 Business Central. Enter your credentials and log in. 8. If prompted, grant permissions for Arc to connect to Dynamics 365 Business Central. You then return to Arc. 9. Click **Add Connection** to finalize the connection. 10. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 11. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Dynamics 365 Business Central, you must choose the action that the Dynamics 365 Business Central connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./dynamics-365-bc-upsert) | Inserts or updates Dynamics 365 Business Central data. By default, if a record already exists in Dynamics 365 Business Central, an update is performed on the existing data in Dynamics 365 Business Central using the values provided from the input. | End | | [Lookup](./dynamics-365-bc-lookup) | Retrieves values from Dynamics 365 Business Central and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Dynamics 365 Business Central. It should be formatted as a SQL query against the Dynamics 365 Business Central tables. | Middle | | [Lookup Stored Procedure](./dynamics-365-bc-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./dynamics-365-bc-select) | Retrieves data from Dynamics 365 Business Central and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./dynamics-365-bc-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./dynamics-365-bc-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Dynamics 365 Business Central Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-lookup Retrieve values from Dynamics 365 Business Central and insert them into an Arc message in the flow. The Lookup action retrieves values from Dynamics 365 Business Central and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Dynamics 365 Business Central connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Dynamics 365 Business Central connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Dynamics 365 Business Central connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Dynamics 365 Business Central connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Dynamics 365 Business Central tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Dynamics 365 Business Central connector can perform Lookup queries against a local cache instead of querying Dynamics 365 Business Central directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Dynamics 365 Business Central on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Dynamics 365 Business Central directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Dynamics 365 Business Central.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Dynamics 365 Business Central Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-lsp Use a stored procedure to retrieve values from Dynamics 365 Business Central and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Dynamics 365 Business Central connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Dynamics 365 Business Central connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Dynamics 365 Business Central connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Dynamics 365 Business Central Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-select Retrieve data from Dynamics 365 Business Central and bring it into Arc. The Select action retrieves data from Dynamics 365 Business Central and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Dynamics 365 Business Central. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Dynamics 365 Business Central, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Dynamics 365 Business Central. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Dynamics 365 Business Central relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Dynamics 365 Business Central supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Dynamics 365 Business Central connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Dynamics 365 Business Central Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Dynamics 365 Business Central\_Select** connector pulls data from a Dynamics 365 Business Central connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Dynamics 365 Business Central\_Select** then passes down the flow from **CSV1**.
# Dynamics 365 Business Central Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-upsert Insert or update Dynamics 365 Business Central data from an Arc message in the flow. The Upsert action updates or inserts Dynamics 365 Business Central data. By default, if a record already exists in Dynamics 365 Business Central, an update is performed on the existing data in Dynamics 365 Business Central using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Dynamics 365 Business Central Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Dynamics 365 Business Central. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Dynamics 365 Business Central connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Dynamics 365 Business Central Upsert connector at the end: ![](/26.4/images/d365bc_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Dynamics 365 Business Central\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Dynamics 365 Business Central.
# Dynamics 365 Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365 Connect Arc to Dynamics 365 for data integration and synchronization The Dynamics 365 connector allows you to integrate Dynamics 365 into your data flow by pushing or pulling data from Dynamics 365. ## Key Capabilities * Microsoft Dynamics 365 integration with multiple Azure authentication schemes * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Dynamics 365. ## Establish a Connection To allow Arc to use data from Dynamics 365, you must first establish a connection to Dynamics 365. There are two ways to establish this connection: * Add a Dynamics 365 connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Dynamics 365**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Dynamics 365 can be re-used across multiple Dynamics 365 connectors. Follow these steps to finish creating your connection: 1. Enter the following connection information in the **Add Connection** modal: * **Name** — The static name of the connection. Set a name of your choice. * **Type** — The connector name. This field is always set to Dynamics 365. * **Organizational URL** — The URL to your Microsoft Dynamics 365 organization. * **Edition** — The Microsoft Dynamics 365 edition. The following editions are supported: CustomerService, FieldService, FinOpsOnline, FinOpsOnPremise, HumanResources, Marketing, ProjectOperations, and Sales. * **Auth Scheme** — The auth scheme to use for the connection. Options are **AzureAD**, **AzureServicePrincipal**, and **AzureMSI**. * **Azure Tenant** — The tenant Id of your Microsoft Azure Active Directory. * **OAuth Grant Type** — The grant type to use to get tokens (AzureAD and AzureServicePrincipal auth schemes only.) Options are **Code**, **Client**, and **Password**. * **OAuth Client Id** — The client Id for your account. (Client grant type only.) * **OAuth Client Secret** — The client secret for your account. (Client grant type only.) 2. If necessary, click **Advanced** to open the drop-down menu of advanced connection settings. These settings should not be needed in most cases. 3. Click **Connect** to ensure that Arc can connect to Dynamics 365 with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. Select the newly created connection from the **Connection** drop-down list that appears on the **Settings** tab of the **Configuration** pane. 6. Click **Save Changes**. ## Select an Action After establishing a connection to Dynamics 365, you must choose the action that the Dynamics 365 connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./dynamics-365-upsert) | Inserts or updates Dynamics 365 data. By default, if a record already exists in Dynamics 365, an update is performed on the existing data in Dynamics 365 using the values provided from the input. | End | | [Lookup](./dynamics-365-lookup) | Retrieves values from Dynamics 365 and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Dynamics 365. It should be formatted as a SQL query against the Dynamics 365 tables. | Middle | | [Lookup Stored Procedure](./dynamics-365-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./dynamics-365-select) | Retrieves data from Dynamics 365 and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./dynamics-365-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./dynamics-365-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Dynamics 365 Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-ac Advanced configuration options for the Dynamics 365 connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Dynamics 365 connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Dynamics 365 connector models Dynamics 365 tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Dynamics 365 values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Dynamics 365 connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Dynamics 365. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Dynamics 365 connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Dynamics 365 connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Dynamics 365. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Dynamics 365, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Dynamics 365. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Dynamics 365 connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Dynamics 365 Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Dynamics 365 connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Dynamics 365 connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Dynamics 365 Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-lookup Retrieve values from Dynamics 365 and insert them into an Arc message in the flow. The Lookup action retrieves values from Dynamics 365 and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Dynamics 365 connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Dynamics 365 connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Dynamics 365 connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Dynamics 365 connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Dynamics 365 tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Dynamics 365 connector can perform Lookup queries against a local cache instead of querying Dynamics 365 directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Dynamics 365 on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Dynamics 365 directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Dynamics 365.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Dynamics 365 Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-lsp Use a stored procedure to retrieve values from Dynamics 365 and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Dynamics 365 connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Dynamics 365 connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Dynamics 365 connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Dynamics 365 Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-select Retrieve data from Dynamics 365 and bring it into Arc. The Select action retrieves data from Dynamics 365 and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Dynamics 365. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Dynamics 365, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Dynamics 365. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Dynamics 365 relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Dynamics 365 supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Dynamics 365 connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Dynamics 365 Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Dynamics 365\_Select** connector pulls data from a Dynamics 365 connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Dynamics 365\_Select** then passes down the flow from **CSV1**.
# Dynamics 365 Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-upsert Insert or update Dynamics 365 data from an Arc message in the flow. The Upsert action updates or inserts Dynamics 365 data. By default, if a record already exists in Dynamics 365, an update is performed on the existing data in Dynamics 365 using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Dynamics 365 Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Dynamics 365. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/dynamics-365/dynamics-365-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Dynamics 365 connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Dynamics 365 Upsert connector at the end: ![](/26.4/images/d365_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Dynamics 365\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Dynamics 365.
# MySQL Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql Connect Arc to MySQL for data integration and synchronization The MySQL connector allows you to integrate MySQL into your data flow by pushing or pulling data from MySQL. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to MySQL. ## Establish a Connection To allow Arc to use data from MySQL, you must first establish a connection to MySQL. There are two ways to establish this connection: * Add a MySQL connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **MySQL**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to MySQL can be reused across multiple MySQL connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to MySQL. * **Server** — The address of the MySQL server you want to connect to. * **Port** — The port to use when connecting to the server. * **Database** — The name of the database you want to target. * **User** — The username to use for logging in. * **Password** — The password for the user entered above. * **AWS Access Key** — Your AWS account access key (only required for AWS authentication). * **AWS Secret Key** — Your AWS account secret key (only required for AWS authentication). * **AWS Role ARN** — The Amazon Resource Name of the role (only required for AWS authentication). 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to MySQL with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to MySQL, you must choose the action that the MySQL connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./mysql-upsert) | Inserts or updates MySQL data. By default, if a record already exists in MySQL, an update is performed on the existing data in MySQL using the values provided from the input. | End | | [Lookup](./mysql-lookup) | Retrieves values from MySQL and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from MySQL. It should be formatted as a SQL query against the MySQL tables. | Middle | | [Lookup Stored Procedure](./mysql-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./mysql-select) | Retrieves data from MySQL and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./mysql-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./mysql-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received. | | FileFormat | Evaluates to the output file format (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # MySQL Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-ac Advanced configuration options for the MySQL connector in Arc. The **Advanced** tab allows you to configure advanced properties of the MySQL connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The MySQL connector models MySQL tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and MySQL values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The MySQL connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into MySQL. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The MySQL connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the MySQL connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by MySQL. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by MySQL, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from MySQL. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure MySQL connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # MySQL Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-esp Execute a stored procedure in MySQL from an Arc message in the flow. The Execute Stored Procedures action treats data coming into the MySQL connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the MySQL connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # MySQL Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-lookup Retrieve values from MySQL and insert them into an Arc message in the flow. The Lookup action retrieves values from MySQL and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the MySQL connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the MySQL connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the MySQL connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the MySQL connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from MySQL tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The MySQL connector can perform Lookup queries against a local cache instead of querying MySQL directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query MySQL on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries MySQL directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to MySQL.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # MySQL Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-lsp Execute a MySQL stored procedure to look up and insert values into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the MySQL connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the MySQL connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the MySQL connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# MySQL Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-select Retrieve records from MySQL and use them to generate Arc messages in the flow. The Select action retrieves data from MySQL and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries MySQL. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to MySQL, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from MySQL. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/mysql/mysql-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in MySQL relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what MySQL supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured MySQL connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a MySQL Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/mysql_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **MySQL\_Select** connector pulls data from a MySQL connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **MySQL\_Select** then passes down the flow from **CSV1**.
# MySQL Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mysql/mysql-upsert Insert or update records in MySQL from an Arc message in the flow. The Upsert action updates or inserts MySQL data. By default, if a record already exists in MySQL, an update is performed on the existing data in MySQL using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The MySQL Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from MySQL. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/mysql/mysql-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured MySQL connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a MySQL Upsert connector at the end: ![](/26.4/images/mysql_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **MySQL\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to MySQL.
# Oracle Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle Connect Arc to Oracle for data integration and synchronization The Oracle connector allows you to integrate Oracle into your data flow by pushing or pulling data from Oracle. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to Oracle. ## Prerequisites Before using an Oracle connector, you must already have a live Oracle database server and a clear network path to that server from Arc. You must also install and configure the dependencies that Oracle requires. Follow the directions below for your version of Arc. ### .NET Edition The .NET edition requires the Oracle Data Provider for .NET-managed driver (ODP.NET), which Oracle distributes through NuGet. ODP.NET is not self-contained: it depends on 11 assemblies, and those assemblies were compiled against different versions of one another. You therefore need to install all 12 files and add binding redirects so that .NET resolves them consistently. Install the exact versions listed below. They are the set that Oracle.ManagedDataAccess 23.26.301 resolves and that CData tests as a unit. Mixing in other versions requires changing the binding redirects to match. In the steps that follow, `[InstallationDirectory]` refers to the Arc installation directory (by default, `C:\Program Files\CData\CData Arc`). > [!NOTE] > > Oracle.ManagedDataAccess 23.26.301 targets .NET Framework 4.7.2. Confirm that the Arc server has .NET Framework 4.7.2 or later before you begin. #### Step 1: Download the Packages Use the NuGet CLI to download Oracle.ManagedDataAccess and everything it depends on in one step. 1. Download the NuGet CLI from [Microsoft](https://dist.nuget.org/win-x86-commandline/v7.9.0/nuget.exe) and save it to an empty working directory, such as `C:\Temp\OracleProvider`. 2. Open a command prompt in that directory and run the following command. Replace `C:\Temp\OracleProvider` with the directory where you saved `nuget.exe`. ``` "C:\Temp\OracleProvider\nuget.exe" install Oracle.ManagedDataAccess -Version 23.26.301 -OutputDirectory .\packages -Source https://api.nuget.org/v3/index.json -Framework net472 -DependencyVersion Lowest -NonInteractive ``` This downloads Oracle.ManagedDataAccess and its transitive dependencies from the official NuGet feed into a `packages` subfolder, with one folder per package. > [!WARNING] > > Run `nuget.exe` by its full path. Calling it by name alone runs whichever copy appears first on your `PATH`, which might be an older release. This command requires NuGet CLI 4.4 or later. Earlier releases report `Unknown option: '-Framework'` or `Unknown option: '-DependencyVersion'`, and releases before 4.1 cannot connect to nuget.org, reporting `Could not create SSL/TLS secure channel` followed by a misleading message that the package version cannot be found. The command resolves the 12 packages below. Take one assembly from each, using the path shown in the **File in the package** column, and confirm that the versions match.
Package Version File in the package Assembly version
Oracle.ManagedDataAccess 23.26.301 `lib\net472\Oracle.ManagedDataAccess.dll` 4.122.23.1
Microsoft.Bcl.AsyncInterfaces 8.0.0 `lib\net462\Microsoft.Bcl.AsyncInterfaces.dll` 8.0.0.0
System.Buffers 4.6.1 `lib\net462\System.Buffers.dll` 4.0.5.0
System.Collections.Immutable 10.0.3 `lib\net462\System.Collections.Immutable.dll` 10.0.0.3
System.Diagnostics.DiagnosticSource 6.0.2 `lib\net461\System.Diagnostics.DiagnosticSource.dll` 6.0.0.2
System.Formats.Asn1 8.0.1 `lib\net462\System.Formats.Asn1.dll` 8.0.0.1
System.Memory 4.6.3 `lib\net462\System.Memory.dll` 4.0.5.0
System.Numerics.Vectors 4.6.1 `lib\net462\System.Numerics.Vectors.dll` 4.1.6.0
System.Runtime.CompilerServices.Unsafe 6.1.2 `lib\net462\System.Runtime.CompilerServices.Unsafe.dll` 6.0.3.0
System.Text.Encodings.Web 8.0.0 `lib\net462\System.Text.Encodings.Web.dll` 8.0.0.0
System.Text.Json 8.0.5 `lib\net462\System.Text.Json.dll` 8.0.0.5
System.Threading.Tasks.Extensions 4.6.0 `lib\net462\System.Threading.Tasks.Extensions.dll` 4.2.1.0
Notice that the package version and the assembly version differ. For example, the System.Memory 4.6.3 package contains assembly version 4.0.5.0. The assembly version is what the binding redirects in [Step 4](#step-4-update-web-config) refer to. > [!NOTE] > > `Oracle.ManagedDataAccessDTC.dll` is not required. Oracle needs it only for distributed transactions on .NET Framework 4.5.1 and earlier, the 23.x package does not ship it, and Arc does not use distributed transactions. If you authenticate through MIT Kerberos, also take `bin\x64\Oracle.ManagedDataAccessIOP.dll` from the Oracle.ManagedDataAccess package and copy it alongside the others in [Step 3](#step-3-copy-the-assemblies). The package supplies only the 64-bit version of this assembly. It is not needed for username and password authentication. > [!NOTE] > > For an Arc server without internet access, run the command above on a machine that does have access and copy the resulting `packages` folder to the Arc server. If you cannot run `nuget.exe` at all, download each package individually from its page on nuget.org instead: a `.nupkg` file is a zip archive, so rename it to `.zip` and extract it. #### Step 2: Stop Arc and Back Up 1. Stop Arc via the Windows service or the CData Arc desktop application. 2. Back up `[InstallationDirectory]\www\bin` and `[InstallationDirectory]\www\Web.config`. [Step 3](#step-3-copy-the-assemblies) overwrites files that Arc ships, and [Step 4](#step-4-update-web-config) edits the configuration that Arc starts from. #### Step 3: Copy the Assemblies Copy the 12 assemblies from [Step 1](#step-1-download-the-packages) into `[InstallationDirectory]\www\bin`, replacing any files that already exist. > [!NOTE] > > Arc ships four of these assemblies for the Apache Kafka connector: System.Buffers, System.Memory, System.Numerics.Vectors, and System.Runtime.CompilerServices.Unsafe. In Arc 26.4 and later, the shipped files are already the versions listed in Step 1, so replacing them changes nothing. In Arc 26.3 and earlier, the shipped files are older than ODP.NET requires and must be replaced. The Kafka connector keeps working in both cases, because the binding redirects in Step 4 route its older references to the new files. #### Step 4: Update Web.config Open `[InstallationDirectory]\www\Web.config` in a text or code editor. This is an XML-formatted file. Make the three changes below. 1. **Register the configuration section handler.** `` must be the first child of ``. If the element does not exist, add it directly after the opening `` tag: ```xml theme={null}
``` 2. **Add the binding redirects.** Navigate to the element `//`, which already contains a `` element. Leave `` in place and add the following after it, inside the same `` element: ```xml theme={null} ``` > [!WARNING] > > Keep exactly one `` element per assembly name. If an entry for one of these assemblies already exists, from an earlier attempt or another connector, replace it instead of adding a second one. 3. **Register the provider.** Navigate to the element `//`. At the bottom of this element, add the following text: ```xml theme={null} ``` 4. Save `Web.config` and close it. #### Step 5: Restart Arc and Verify Start Arc, add an Oracle connector to a flow, and confirm the following two things in order: 1. **Test Connection** succeeds. 2. You can **select a table** in the connector configuration. > [!NOTE] > > Test Connection alone does not confirm that the installation is correct. .NET Framework resolves assembly references the first time they are used, and the code paths that need the binding redirects do not run until Arc reads table metadata. An installation that is missing redirects passes Test Connection, then fails when you select a table. Always complete both checks. If Arc also uses the Apache Kafka connector, test an existing Kafka connection as well, because [Step 3](#step-3-copy-the-assemblies) replaced four assemblies that Kafka shares. #### Upgrading Arc Upgrading Arc overwrites `[InstallationDirectory]\www\bin` but preserves `Web.config`. If an upgrade or reinstall restores older copies of the four shared assemblies, the binding redirects point at versions that are no longer present and both the Oracle and Kafka connectors can fail. Repeat [Step 3](#step-3-copy-the-assemblies) after every upgrade, then confirm that the assembly versions in `[InstallationDirectory]\www\bin` still match the values listed in [Step 1](#step-1-download-the-packages). Upgrading to Arc 26.4 or later installs the same versions the redirects expect, so no action is needed on that path. #### Troubleshooting **The Oracle connector does not appear in the connector list**: One or more of the 12 assemblies is missing from `[InstallationDirectory]\www\bin`. Arc hides the connector when it cannot load the ODP.NET provider factory, without reporting an error. Confirm that all 12 files are present and restart Arc. **Selecting a table fails with `Could not load file or assembly ... 0x80131040`**: A binding redirect is missing or points at the wrong version. The message names the assembly and the version that was requested. Confirm that the version of that assembly in `[InstallationDirectory]\www\bin` matches the assembly version listed for it in [Step 1](#step-1-download-the-packages), and that the `newVersion` of its binding redirect uses that same value. If they do not match, correct the redirect so that `newVersion` is the installed assembly version and `oldVersion` is `0.0.0.0-`. Then restart Arc, because `Web.config` changes only take effect on startup. ### Java Edition 1. Download the Oracle JDBC .jar file from [the Oracle website](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html). * If you are using the Arc embedded Jetty server, select the latest release of **ojdbc11.jar**. * If you are using another hosting option, select the latest release of either **ojdbc11.jar** or **ojdbc8.jar**, depending on the version of JDK your hosting server uses. 2. Move the file to the appropriate location: * If you are using the embedded Jetty server, move the file to the `lib` folder in the Arc installation directory (`[InstallationDirectory]/lib`). If this folder does not exist, create it. * If you are using another server, move the file to the `lib` folder for the server. If this folder does not exist, create it. ## Establish a Connection To allow Arc to use data from Oracle, you must first establish a connection to Oracle. There are two ways to establish this connection: * Add a Oracle connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Oracle**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Oracle can be re-used across multiple Oracle connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — the static name of the connection. Set this as desired. * **Type** — this is always set to Oracle. * **Connection Type** — the connection type (SID or Service Name) to use. * **User** — the username to use for logging in. * **Password** — the password for the user entered above. * **Server** — the address of the Oracle server you want to connect to. * **Port** — the port to use when connecting to the server. * **SID/Service** — the SID or Service (depending on Connection Type chosen) to use when connecting. * **Other** — other authentication information if needed. 2. If needed, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 3. Click **Test Connection** to ensure that Arc can connect to Oracle with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. ## Select an Action After establishing a connection to Oracle, you must choose the action that the Oracle connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./oracle-upsert) | Inserts or updates Oracle data. By default, if a record already exists in Oracle, an update is performed on the existing data in Oracle using the values provided from the input. | End | | [Lookup](./oracle-lookup) | Retrieves values from Oracle and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Oracle. It should be formatted as a SQL query against the Oracle tables. | Middle | | [Lookup Stored Procedure](./oracle-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./oracle-select) | Retrieves data from Oracle and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./oracle-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./oracle-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Oracle Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-ac Reference for advanced settings available on the Oracle connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the Oracle connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Oracle connector models Oracle tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Oracle values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Oracle connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Oracle. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Oracle connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Oracle connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Oracle. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Oracle, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Oracle. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Oracle connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Oracle Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-esp Use a stored procedure to process data coming into the Oracle connector and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Oracle connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Oracle connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Oracle Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-lookup Retrieve values from Oracle and insert them into an Arc message in the flow. The Lookup action retrieves values from Oracle and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Oracle connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Oracle connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Oracle connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Oracle connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Oracle tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Oracle connector can perform Lookup queries against a local cache instead of querying Oracle directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Oracle on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Oracle directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Oracle.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Oracle Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-lsp Use a stored procedure to look up Oracle data and insert the result into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Oracle connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Oracle connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Oracle connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Oracle Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-select Retrieve data from Oracle and bring it into Arc. The Select action retrieves data from Oracle and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Oracle. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Oracle, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Oracle. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/oracle/oracle-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Oracle relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Oracle supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Oracle connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Oracle Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/oracle_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **Oracle\_Select** connector pulls data from a Oracle connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Oracle\_Select** then passes down the flow from **CSV1**.
# Oracle Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/oracle/oracle-upsert Insert or update data in Oracle from an Arc message in the flow. The Upsert action updates or inserts Oracle data. By default, if a record already exists in Oracle, an update is performed on the existing data in Oracle using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Oracle Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Oracle. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/oracle/oracle-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Oracle connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Oracle Upsert connector at the end: ![](/26.4/images/oracle_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Oracle\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Oracle.
# Other Connectors Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/other-connectors The CData Arc core feature set is extended by additional connectors, most of which are licensed separately from the application itself. Select the name of a connector from the navigation pane to access documentation for that connector. # PostgreSQL Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql Connect Arc to PostgreSQL for data integration and synchronization The PostgreSQL connector allows you to integrate PostgreSQL into your data flow by pushing or pulling data from PostgreSQL. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to PostgreSQL. ## Establish a Connection To allow Arc to use data from PostgreSQL, you must first establish a connection to PostgreSQL. There are two ways to establish this connection: * Add a PostgreSQL connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **PostgreSQL**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to PostgreSQL can be reused across multiple PostgreSQL connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to PostgreSQL. * **Server** — The address of the PostgreSQL server you want to connect to. * **Auth Scheme** — The authentication scheme to use for the connection. Options are **Password**, **AzureAD**, **AzureServicePrincipal**, **AzureServicePrincipalCert**, **AzurePassword**, **AzureMSI**, **AwsIAMRoles**, **AwsEC2Roles**, and **GCPServiceAccount**. Additional fields appear based on your selection. If you leave this blank, the default authentication is used. * **Database** — The name of the database you want to target. * **Port** — The port to use when connecting to the server. * **User** — The username to use for logging in. * **Password** — The password for the user entered above. Only required when your auth scheme is **Password** or **AzurePassword**. * **AWS Access Key** — Your AWS account access key. Obtain it from your AWS security credentials page. Only required when using an AWS-specific authentication method. * **AWS Secret Key** — Your AWS account secret key. Obtain it from your AWS security credentials page. Only required when using an AWS-specific authentication method. * **AWS Role ARN** — The Amazon Resource Name of the role to use when authenticating. Only required when using an AWS-specific authentication method. * **AWS External Id** — The AWS external Id for the role entered above. Only required when using an AWS-specific authentication method. * **Azure Tenant** — The Azure tenant on which you are hosting the database. Only required when using an Azure-specific authentication method. * **OAuth Client Id** — The client Id (also known as the consumer key) assigned to your custom OAuth application. Only required when your auth scheme is **AzureServicePrincipal**. * **OAuth Client Secret** — The client secret assigned to your custom OAuth application. Only required when your auth scheme is **AzureServicePrincipal**. * **OAuth JWT Cert** — The name of the client certificate's JWT certificate store. Only required when your auth scheme is **AzureServicePrincipalCert** or **GCPServiceAccount**. * **OAuth JWT Cert Type** — The type of key store containing the JWT Certificate. Only required when your auth scheme is **AzureServicePrincipalCert** or **GCPServiceAccount**. * **OAuth JWT Cert Password** — The password for the OAuth JWT certificate used to access a password-protected certificate store. If the certificate store does not require a password, leave this blank. Only required when your auth scheme is **AzureServicePrincipalCert** or **GCPServiceAccount**. * **OAuth JWT Cert Subject** — The subject of the OAuth JWT certificate used to locate a matching certificate in the store. Supports partial matches and the wildcard '\*' to select the first certificate. Only required when your auth scheme is **AzureServicePrincipalCert** or **GCPServiceAccount**. * **Scope** — Specifies the scope of the authenticating user's access to the application, to ensure that they get appropriate access to data. Only required when using an Azure-specific authentication method. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to PostgreSQL with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to PostgreSQL, you must choose the action that the PostgreSQL connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./postgresql-upsert) | Inserts or updates PostgreSQL data. By default, if a record already exists in PostgreSQL, an update is performed on the existing data in PostgreSQL using the values provided from the input. | End | | [Lookup](./postgresql-lookup) | Retrieves values from PostgreSQL and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from PostgreSQL. It should be formatted as a SQL query against the PostgreSQL tables. | Middle | | [Lookup Stored Procedure](./postgresql-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./postgresql-select) | Retrieves data from PostgreSQL and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./postgresql-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./postgresql-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # PostgreSQL Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-ac Reference for advanced settings available on the PostgreSQL connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the PostgreSQL connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The PostgreSQL connector models PostgreSQL tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and PostgreSQL values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The PostgreSQL connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into PostgreSQL. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The PostgreSQL connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the PostgreSQL connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by PostgreSQL. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by PostgreSQL, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from PostgreSQL. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure PostgreSQL connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # PostgreSQL Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the PostgreSQL connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the PostgreSQL connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # PostgreSQL Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-lookup Retrieve values from PostgreSQL and insert them into an Arc message in the flow. The Lookup action retrieves values from PostgreSQL and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the PostgreSQL connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the PostgreSQL connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the PostgreSQL connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the PostgreSQL connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from PostgreSQL tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The PostgreSQL connector can perform Lookup queries against a local cache instead of querying PostgreSQL directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query PostgreSQL on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries PostgreSQL directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to PostgreSQL.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # PostgreSQL Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-lsp Use a stored procedure to retrieve values from PostgreSQL and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the PostgreSQL connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the PostgreSQL connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the PostgreSQL connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# PostgreSQL Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-select Retrieve data from PostgreSQL and bring it into Arc to start or continue a flow. The Select action retrieves data from PostgreSQL and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries PostgreSQL. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to PostgreSQL, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from PostgreSQL. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/postgresql/postgresql-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in PostgreSQL relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what PostgreSQL supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured PostgreSQL connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a PostgreSQL Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/postgresql_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **PostgreSQL\_Select** connector pulls data from a PostgreSQL connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **PostgreSQL\_Select** then passes down the flow from **CSV1**.
# PostgreSQL Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/postgresql/postgresql-upsert Insert or update data in PostgreSQL from an Arc message in the flow. The Upsert action updates or inserts PostgreSQL data. By default, if a record already exists in PostgreSQL, an update is performed on the existing data in PostgreSQL using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The PostgreSQL Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from PostgreSQL. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/postgresql/postgresql-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured PostgreSQL connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a PostgreSQL Upsert connector at the end: ![](/26.4/images/postgresql_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **PostgreSQL\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to PostgreSQL.
# SQL Server Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server Connect Arc to SQL Server for data integration and synchronization The SQL Server connector allows you to integrate SQL Server into your data flow by pushing or pulling data from SQL Server. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to SQL Server. ## Establish a Connection To allow Arc to use data from SQL Server, you must first establish a connection to SQL Server. There are two ways to establish this connection: * Add a SQL Server connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **SQL Server**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to SQL Server can be re-used across multiple SQL Server connectors. ### Enter Connection Settings 1. Enter the connection information: * **Name** — the static name of the connection. * **Type** — this is always set to SQL Server. * **Auth Scheme** — the authentication scheme to use for the connection. * **Server** — the URL of the SQL server. * **Port** — the port that the SQL server is listening on. * **Database** — the database to connect to. 2. Enter the connection information for the authentication scheme you selected above: * **Password**/**NTLM** — enter your login credentials in the **User** and **Password** fields. * **Kerberos** — enter values for **User**, **Password**, **Kerberos KDC**, and **Kerberos SPN**. * **Azure Password** — enter your login credentials in the **User** and **Password** fields, and enter your tenant in the **Azure Tenant** field. * **Azure AD**/**Azure Service Principal** — enter your tenant in the **Azure Tenant** field, and enter your OAuth information in the **OAuth Client Id** and **OAuth Client Secret** fields. * **Azure MSI** — no additional information needed. 3. If needed, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 4. Click **Test Connection** or **Connect** to ensure that Arc can connect to SQL Server with the provided information. If an error occurs, check all fields and try again. 5. Click **Add Connection** to finalize the connection. 6. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 7. Click **Save Changes**. ## Select an Action After establishing a connection to SQL Server, you must choose the action that the SQL Server connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./sql-server-upsert) | Inserts or updates SQL Server data. By default, if a record already exists in SQL Server, an update is performed on the existing data in SQL Server using the values provided from the input. | End | | [Lookup](./sql-server-lookup) | Retrieves values from SQL Server and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from SQL Server. It should be formatted as a SQL query against the SQL Server tables. | Middle | | [Lookup Stored Procedure](./sql-server-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./sql-server-select) | Retrieves data from SQL Server and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./sql-server-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./sql-server-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # SQL Server Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-ac Advanced configuration options for the SQL Server connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the SQL Server connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The SQL Server connector models SQL Server tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and SQL Server values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The SQL Server connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into SQL Server. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The SQL Server connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the SQL Server connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by SQL Server. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by SQL Server, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from SQL Server. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure SQL Server connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # SQL Server Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the SQL Server connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the SQL Server connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # SQL Server Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-lookup Retrieve values from SQL Server and insert them into an Arc message in the flow. The Lookup action retrieves values from SQL Server and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the SQL Server connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the SQL Server connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the SQL Server connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the SQL Server connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from SQL Server tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The SQL Server connector can perform Lookup queries against a local cache instead of querying SQL Server directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query SQL Server on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries SQL Server directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to SQL Server.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # SQL Server Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-lsp Use a stored procedure to retrieve values from SQL Server and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the SQL Server connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the SQL Server connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the SQL Server connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# SQL Server Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-select Retrieve data from SQL Server and bring it into Arc. The Select action retrieves data from SQL Server and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries SQL Server. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to SQL Server, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from SQL Server. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/sql-server/sql-server-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in SQL Server relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what SQL Server supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured SQL Server connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a SQL Server Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/sqlserver_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **SQL Server\_Select** connector pulls data from a SQL Server connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **SQL Server\_Select** then passes down the flow from **CSV1**.
# SQL Server Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sql-server/sql-server-upsert Insert or update SQL Server data from an Arc message in the flow. The Upsert action updates or inserts SQL Server data. By default, if a record already exists in SQL Server, an update is performed on the existing data in SQL Server using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The SQL Server Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from SQL Server. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/sql-server/sql-server-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured SQL Server connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a SQL Server Upsert connector at the end: ![](/26.4/images/sqlserver_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **SQL Server\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to SQL Server.
# SQLite Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite Connect Arc to SQLite for data integration and synchronization The SQLite connector allows you to integrate SQLite into your data flow by pushing or pulling data from SQLite. > [!NOTE] > > This connector is only available in the [.NET edition](/26.4/self-hosted/en/getting-started/installation/windows-edition) of Arc. ## Key Capabilities * Full-featured database connectivity with platform-specific optimizations * Standard database operations (Select, Lookup, Upsert) with transaction support * Advanced connection pooling and performance tuning capabilities * Support for stored procedures, functions, and complex query operations Follow the steps below to connect Arc to SQLite. ## Establish a Connection There are two ways to establish this connection: * Add a SQLite connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **SQLite**, and click **Next**. > [!NOTE] > > Connections to SQLite can be reused across multiple SQLite connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name**: The static name of the connection. * **Type**: This is always set to SQLite. * **Data Source**: The local file path to the database file. For example, `C:\ProgramData\CData\Arc\db\app_logs.db`. * **Other**: A field where you can specify non-standard options for the driver. (Only applicable when you are using the cross-platform edition.) 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can reach the specified file. If an error occurs, check the path and folder permissions. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to SQLite, you must choose the action that the SQLite connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./sqlite-upsert) | Inserts or updates SQLite data. By default, if a record already exists in SQLite, an update is performed on the existing data in SQLite using the values provided from the input. | End | | [Lookup](./sqlite-lookup) | Retrieves values from SQLite and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from SQLite. It should be formatted as a SQL query against the SQLite tables. | Middle | | [Lookup Stored Procedure](./sqlite-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./sqlite-select) | Retrieves data from SQLite and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./sqlite-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./sqlite-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # SQLite Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-ac Configure advanced settings for the SQLite connector in Arc. The **Advanced** tab allows you to configure advanced properties of the SQLite connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The SQLite connector models SQLite tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and SQLite values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The SQLite connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into SQLite. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The SQLite connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the SQLite connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by SQLite. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by SQLite, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from SQLite. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure SQLite connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # SQLite Execute Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-esp Execute a stored procedure against a SQLite database from an Arc message in the flow. The Execute Stored Procedures action treats data coming into the SQLite connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the SQLite connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # SQLite Lookup Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-lookup Retrieve values from SQLite and insert them into an Arc message in the flow. The Lookup action retrieves values from SQLite and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the SQLite connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the SQLite connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the SQLite connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the SQLite connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from SQLite tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The SQLite connector can perform Lookup queries against a local cache instead of querying SQLite directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query SQLite on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries SQLite directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to SQLite.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # SQLite Lookup Stored Procedure Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-lsp Execute a stored procedure to retrieve values from SQLite and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the SQLite connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the SQLite connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the SQLite connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# SQLite Select Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-select Select records from a SQLite database and pass them into the flow as Arc messages. The Select action retrieves data from SQLite and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries SQLite. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to SQLite, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from SQLite. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/sqlite/sqlite-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in SQLite relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what SQLite supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured SQLite connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a SQLite Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/sqlite_xmlflow_select.png) Data in this example flow passes through these steps:
  1. The **SQLite\_Select** connector pulls data from a SQLite connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **SQLite\_Select** then passes down the flow from **CSV1**.
# SQLite Upsert Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sqlite/sqlite-upsert Insert or update records in a SQLite database from an Arc message in the flow. The Upsert action updates or inserts SQLite data. By default, if a record already exists in SQLite, an update is performed on the existing data in SQLite using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The SQLite Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from SQLite. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/sqlite/sqlite-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured SQLite connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a SQLite Upsert connector at the end: ![](/26.4/images/sqlite_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **SQLite\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to SQLite.
# Dynamics 365 Business Central Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-ac Advanced configuration options for the Dynamics 365 Business Central connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Dynamics 365 Business Central connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Dynamics 365 Business Central connector models Dynamics 365 Business Central tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Dynamics 365 Business Central values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Dynamics 365 Business Central connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Dynamics 365 Business Central. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Dynamics 365 Business Central connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Dynamics 365 Business Central connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Dynamics 365 Business Central. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Dynamics 365 Business Central, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Dynamics 365 Business Central. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Dynamics 365 Business Central connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Dynamics 365 Business Central Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/dynamics-365-bc/dynamics-365-bc-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Dynamics 365 Business Central connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Dynamics 365 Business Central connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Exact Online Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online Connect Arc to Exact Online for data integration and synchronization The Exact Online connector allows you to integrate Exact Online into your data flow by pushing or pulling data from Exact Online. ## Key Capabilities * Cloud-based ERP software integration with OAuth authentication for accounting, CRM, and project management allowing for pushing and pulling data from Exact Online * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Exact Online. ## Establish a Connection To allow Arc to use data from Exact Online, you must first establish a connection to Exact Online. There are two ways to establish this connection: * Add a Exact Online connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Exact Online**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Exact Online can be re-used across multiple Exact Online connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter a name for the connection. 2. In a new browser tab or window, open Exact Online. 3. If you have not already done so, create an Exact Online developer account. 4. Open the App Center and click **Manage Apps** > **Add a New Application**. 5. Set the value for **Redirect URI** to a value in this format: *\[baseURL]/src/oauthCallback.rst* * The value for *\[baseURL]* is the URL that you use to access Arc. For example, *[https://localhost:8080](https://localhost:8080)* or *[https://myArcServer.com](https://myArcServer.com)* * The complete URI might look something like *[https://localhost:8080/src/oauthCallback.rst](https://localhost:8080/src/oauthCallback.rst)* 6. Save your application, then click **Edit** to display the client Id and client secret. Copy these values. 7. Return to Arc. Select your **Region**, and enter your **OAuth Client Id** and **OAuth Client Secret** values in the connection modal. 8. If needed, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 9. Click **Connect**. If you are not still logged into your Exact Online account, log in. 10. If prompted, grant permissions for Arc to connect to Exact Online. You are then returned to Arc. 11. Click **Add Connection** to finalize the connection. 12. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 13. Click **Save Changes**. ## Select an Action After establishing a connection to Exact Online, you must choose the action that the Exact Online connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./exact-online-upsert) | Inserts or updates Exact Online data. By default, if a record already exists in Exact Online, an update is performed on the existing data in Exact Online using the values provided from the input. | End | | [Lookup](./exact-online-lookup) | Retrieves values from Exact Online and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Exact Online. It should be formatted as a SQL query against the Exact Online tables. | Middle | | [Lookup Stored Procedure](./exact-online-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./exact-online-select) | Retrieves data from Exact Online and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./exact-online-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./exact-online-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Exact Online Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-ac Advanced configuration options for the Exact Online connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Exact Online connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Exact Online connector models Exact Online tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Exact Online values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Exact Online connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Exact Online. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Exact Online connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Exact Online connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Exact Online. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Exact Online, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Exact Online. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Exact Online connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Exact Online Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Exact Online connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Exact Online connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Exact Online Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-lookup Retrieve values from Exact Online and insert them into an Arc message in the flow. The Lookup action retrieves values from Exact Online and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Exact Online connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Exact Online connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Exact Online connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Exact Online connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Exact Online tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Exact Online connector can perform Lookup queries against a local cache instead of querying Exact Online directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Exact Online on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Exact Online directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Exact Online.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Exact Online Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-lsp Use a stored procedure to retrieve values from Exact Online and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Exact Online connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Exact Online connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Exact Online connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Exact Online Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-select Retrieve data from Exact Online and bring it into Arc. The Select action retrieves data from Exact Online and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Exact Online. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Exact Online, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Exact Online. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/exact-online/exact-online-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Exact Online relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Exact Online supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Exact Online connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Exact Online Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Exact Online\_Select** connector pulls data from a Exact Online connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Exact Online\_Select** then passes down the flow from **CSV1**.
# Exact Online Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/exact-online/exact-online-upsert Insert or update Exact Online data from an Arc message in the flow. The Upsert action updates or inserts Exact Online data. By default, if a record already exists in Exact Online, an update is performed on the existing data in Exact Online using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Exact Online Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Exact Online. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/exact-online/exact-online-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Exact Online connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Exact Online Upsert connector at the end: ![](/26.4/images/exactonline_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Exact Online\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Exact Online.
# FHIR Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir Connect Arc to FHIR for data integration and synchronization The FHIR connector allows you to integrate FHIR into your data flow by pushing or pulling data from FHIR. ## Key Capabilities * Fast Healthcare Interoperability Resources (FHIR) standard support for compliant healthcare data exchange * RESTful API integration with healthcare-specific data formats for patient records, observations, and clinical data * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to FHIR. ## Establish a Connection To allow Arc to use data from FHIR, you must first establish a connection to FHIR. There are two ways to establish this connection: * Add a FHIR connector to your flow. Then, in the settings pane, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **FHIR**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to FHIR can be re-used across multiple FHIR connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter the connection information: * **Name** — The static name of the connection. Set this as desired. * **Type** — This is always set to FHIR. * **Connection Type** — The FHIR API implementation that you want to use. Options are **Generic** (for any custom implementation of the FHIR API), **Azure**, **AWS**, and **Google**. * **Auth Scheme** — The authentication scheme to use for the connection. Options are **None** (the default option), **OAUTH**, **Azure AD**, and **AwsRootKeys**. * **URL** — The URL to the FHIR server that you want to use. 2. Create the connection according to your selected Auth scheme: * **None** — No authentication is used. * **OAUTH** — Enter your OAuth client Id and the OAuth client secret. * **AzureAD** — Enter your OAuth client Id and OAuth client secret. * **AwsRootKeys** — Enter your AWS access key and AWS secret key. 3. If necessary, click **Advanced** to open the drop-down menu of advanced connection settings. These should not be needed in most cases. 4. Click **Test Connection** to ensure that Arc can connect to FHIR with the provided information. If an error occurs, check all fields and try again. 5. Click **Add Connection** to finalize the connection. 6. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 7. Click **Save Changes**. ## Select an Action After establishing a connection to FHIR, you must choose the action that the FHIR connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Lookup](./fhir-lookup) | Retrieves values from FHIR and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from FHIR. It should be formatted as a SQL query against the FHIR tables. | Middle | | [Lookup Stored Procedure](./fhir-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./fhir-select) | Retrieves data from FHIR and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./fhir-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./fhir-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ### Proxy Settings These are a collection of settings that identify and authenticate to the proxy through which the connection should be routed. By default, this section uses the global settings on the [Proxy Settings](/26.4/self-hosted/en/getting-started/administration/settings/proxy-settings) portion of the [Security Settings](/26.4/self-hosted/en/getting-started/administration/settings/security) page. Clear the checkbox to supply settings specific to your connector.
SettingDescription
**Proxy Type** The protocol used by a proxy-based firewall.
**Proxy Host** The name or IP address of a proxy-based firewall.
**Proxy Port** The TCP port for a proxy-based firewall.
**Proxy User** The user name to use to authenticate with a proxy-based firewall.
**Proxy Password** A password used to authenticate to a proxy-based firewall.
**Authentication Scheme** Leave the default **None** or choose from one of the following authentication schemes: **Basic**, **Digest**, **Proprietary**, or **NTLM**.
## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # FHIR Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir-ac Advanced configuration options for the FHIR connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the FHIR connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The FHIR connector models FHIR tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and FHIR values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The FHIR connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into FHIR. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The FHIR connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the FHIR connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by FHIR. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by FHIR, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from FHIR. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure FHIR connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # FHIR Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the FHIR connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the FHIR connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # FHIR Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir-lookup Retrieve values from FHIR and insert them into an Arc message in the flow. The Lookup action retrieves values from FHIR and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the FHIR connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the FHIR connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the FHIR connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the FHIR connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from FHIR tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The FHIR connector can perform Lookup queries against a local cache instead of querying FHIR directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query FHIR on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries FHIR directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to FHIR.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # FHIR Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir-lsp Use a stored procedure to retrieve values from FHIR and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the FHIR connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the FHIR connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the FHIR connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# FHIR Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/fhir/fhir-select Retrieve data from FHIR and bring it into Arc. The Select action retrieves data from FHIR and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries FHIR. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to FHIR, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from FHIR. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/fhir/fhir-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in FHIR relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what FHIR supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured FHIR connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a FHIR Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **FHIR\_Select** connector pulls data from a FHIR connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **FHIR\_Select** then passes down the flow from **CSV1**.
# Finalize Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/finalize The Finalize connector completes a deferred source message after the downstream flow succeeds, so the source connector can safely clean up. The Finalize connector completes a deferred source message. Place it at the end of the success path of a flow to confirm that the flow processed a file successfully, so the connector that received the file can perform its post-processing, such as deleting the source file. The Finalize connector is included with CData Arc and does not require a separate license. > [!NOTE] > > Only the [File connector](./file) can defer finalization. Other source connectors do not support this feature. ## Key Capabilities * Deferred cleanup of source files, so a received file is not deleted until the downstream flow succeeds * Automatic timeout handling that releases pending files for another receive without deleting the source * No configuration required to link to the source connector, because the message carries the information needed to find it * Clear error messages when a message cannot be finalized ## Overview By default, a source connector such as the File connector completes a message as soon as it receives it. Any post-processing, such as deleting the file from the source folder, happens immediately, even if the downstream flow later fails. When you enable **Defer received-file finalization** on the File connector, the connector holds each received file in a pending state instead. The Finalize connector completes the pending file after the rest of the flow succeeds. When a message reaches the Finalize connector, the connector: 1. Reads the deferred source information that the message carries. 2. Finds the source connector that received the file. 3. Tells the source connector to finalize the file, which triggers the source connector's configured cleanup (for example, deleting the file when **Delete files (after received)** is enabled). The Finalize connector has no output. It can only be used as the last connector in a [flow](../flows/flows). ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab *Settings related to the core configuration of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | A toggle that instructs the connector to automatically send files when they are ready. | | **Retry Interval** | The interval the connector waits before retrying a failed send. | | **Max Attempts** | The number of attempts the connector makes to send the message. Setting this value to *1* instructs the connector to only make the initial send attempt without retrying. The connector waits the duration specified by **Retry Interval** between each attempt. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Transactions Tab This tab lists all messages associated with the connector. Use the search bar to find specific messages, or click the funnel icon to apply a filter. You can filter by time, message direction, and/or status. ## Using Finalize with the File Connector To delay deletion of a received file until the flow succeeds: 1. On the File connector that receives the files, enable **Defer received-file finalization** and, if you want the source files removed, **Delete files (after received)**. See [Receive](./file#receive). 2. Build the rest of the flow that processes the file. 3. Add a Finalize connector and connect it to the **Success Path** of the last connector that processes the file. For example, a flow that receives files with a File connector, translates them with an X12 connector, upserts the data into SQL Server, and then finalizes them is configured as follows: ![A flow with a File connector, an X12 connector, a SQL Server connector, and a Finalize connector](https://mintcdn.com/cdata-arc/yfA3o6U6SThtnLN0/26.4/images/finalize-flow.png?fit=max\&auto=format\&n=yfA3o6U6SThtnLN0\&q=85\&s=c5714d55ee8a25aa6bae5c676cb5d908) 1. The File connector receives a file from its configured path and holds it as pending. 2. The X12 connector translates the file to XML. 3. The SQL Server connector upserts the XML data into SQL Server. 4. The SQL Server connector's **Success Path** is connected to the Finalize connector, which completes the pending file. Only the File connector that receives the files at the start of the flow can defer finalization. A File connector later in the flow that sends files to a folder does not defer anything, but the flow can still end with a Finalize connector that completes the original file. The file remains pending, and is excluded from being received again, until one of the following happens: * **The file reaches the Finalize connector.** The File connector completes the file and performs the configured cleanup. * **The pending timeout is reached.** The **Receive Pending Timeout (minutes)** setting on the File connector's [Automation tab](./file#automation-settings) controls this timeout. The message is marked with a Pending Timeout error, the source file is not deleted, and the file is released so that it can be received again. > [!NOTE] > > Messages that do not come from a connector with deferred finalization enabled cannot be finalized. Sending a regular message to the Finalize connector results in an error. ### Source File Changes If the source file changes between the time it is received and the time it is finalized, the connector does not delete it. The application log contains the following message, where `{0}` is the name of the file: `Source file '{0}' changed after it was received. Deferred finalization will not delete it.` ### Pending Timeout The **Receive Pending Timeout (minutes)** setting on the File connector's **Automation** tab controls how long a received file stays pending. The default is 10 minutes, and setting it to 0 keeps files pending indefinitely. If a pending message is not finalized before the timeout elapses: * The application log records that the message exceeded the pending timeout while waiting for finalization. * An error notification is sent with the subject **Send Failure: Pending Timeout**. * The message is marked with a Pending Timeout error. This appears as **Error** in the **Transactions** tab. * The source file is not deleted, and the file becomes eligible to be received again. The pending-message check runs as the **Check Pending** automation action. At the **Debug** [log level](../getting-started/administration/activity#log-level), the [automation log](../getting-started/administration/activity#automation-logs) records each check and the number of timed-out messages it handled. ### Deferred Message Information A file that is held for deferred finalization carries extra information in its message headers. To view the headers, open the message on the **Transactions** tab of the File connector and go to the **Details** tab. | Header | Description | | - | - | | **File-Deferred-Source** | The path of the source file that is waiting to be finalized. | | **Defer-Finalize** | The workspace, connector, and message Id that are part of the finalize chain. The Finalize connector uses this header to find the source connector and the pending message to complete. | | **Processed** | Lists each connector that processed the message and its status. The File connector with deferred finalization is listed as **Pending** until the message is finalized. | ## Troubleshooting The Finalize connector reports the following errors when it cannot complete a message. | Error | Cause and resolution | | - | - | | The message has no deferred source lineage. | The message did not come from a connector with deferred finalization enabled. Enable **Defer received-file finalization** on the File connector that starts the flow. To confirm, check that the message has a **Defer-Finalize** header in its details. | | The deferred source lineage is invalid or has already been completed. | The information that identifies the source message is malformed, or the message was already finalized. Check that the flow does not send the same message to Finalize more than once. | | The deferred source connector `{workspace}:{connector}` does not exist. | The source connector was deleted or renamed after it received the file. Restore the connector, or process the file again from the source. | | The deferred source connector `{workspace}:{connector}` is busy. | The source connector is processing other work. Retry the send after the source connector finishes. | | The pending source message `{id}` does not exist in `{workspace}:{connector}`. | The pending message timed out before it reached the Finalize connector. If this happens regularly, check whether the flow is taking longer than the File connector's **Receive Pending Timeout (minutes)** to complete, and increase the timeout if needed. | # IBM MQ Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/ibmmq Connect Arc to IBM MQ queues through the REST API to send and receive enterprise messages. The IBM MQ connector provides access to IBM MQ queues through the REST API. ## Key Capabilities * Enterprise message queuing through IBM MQ REST API integration * Queue manager connectivity with TLS encryption support * Bidirectional message processing with configurable authentication * Automatic retry logic with configurable intervals and maximum attempts ## Overview IBM MQ connectors are configured with a specific queue to which messages are pushed or from which messages are retrieved. Files that arrive in the **Transactions** tab of the connector are pushed as a message to the queue, and messages pulled from the queue are written as files to the **Transactions** tab. Each connector instance can only be configured with a single queue. ## Prerequisites To establish a connection with IBM MQ, at a minimum users must have: * An IBM MQ server and a clear network path to that server from Arc * A queue manager with at least one active queue > [!NOTE] > > CData does not provide setup assistance or support for the creation or configuration of the IBM MQ server. For questions about the setup of an IBM MQ server, consult the [Introduction to IBM MQ](https://www.ibm.com/docs/en/ibm-mq/9.4?topic=mq-introduction) documentation. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Remote Host** | The hostname or IP address of the system hosting the MQ service. | | **Remote Port** | The port on the remote host where the MQ service is listening. | | **Queue Manager** | The name of the local queue manager for the desired queue. | | **Queue** | The name of the queue to push messages to or pull messages from. | #### TLS Settings *Settings related to transport security.* | Setting | Description | | - | - | | **Use TLS when connecting with IBM MQ servers** | Whether to use TLS encryption as transport security when connecting and communicating with IBM MQ servers. When disabled, all communication is in plain text. | | **Server Public Certificate** | When TLS is enabled, set this field to the TLS/SSL certificate that the IBM MQ presents to verify its identity. Set this to **Any Certificate** to implicitly trust the IBM MQ server. | #### Client Authentication *Settings related to authenticating against the remote service.* | Setting | Description | | - | - | | **Username** | The username credential for authenticating against the remote server. | | **Password** | The password for the specified username. | #### Other Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The following settings are required to establish a connection: * Remote Host * Remote Port * Queue Manager * Queue If client authentication is required, you must also set the following: * Username * Password Use the **Test Connection** button to verify the connection to the server. ## Sending Messages Files that arrive in the **Transactions** tab for the connector are pushed to the configured queue as a message. Files can arrive in this folder either by being placed there directly, or from another connector earlier in the [flow](../flows/flows). You need to set up multiple IBM MQ connectors to push messages to multiple queues. You can use a connector like the [Branch connector](./branch) to route files to the appropriate IBM MQ connector. ## Receiving Messages You can configure the IBM MQ connector to automatically poll the remote queue for messages to download. Use the **Receive Automation** and **Receive** settings on the [Automation tab](#automation-tab). When enabled, the connector waits for the specified interval before attempting to retrieve messages from the queue and pushing them to the Output folder. If the IBM MQ connector is connected to other connectors in a flow, downloaded messages are automatically passed to the next connector. Otherwise, the messages remain in the Output folder. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Infor Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/infor Connect Arc to Infor CloudSuite Industrial to exchange EDI documents with automatic XML and flat file conversion. The Infor connector supports transmitting EDI documents to and from the Infor CloudSuite Industrial (formerly Syteline) platform. ## Key Capabilities * Infor CloudSuite Industrial EDI document exchange with XML to flat file conversion * Bidirectional file transfer with separate demand-side and supply-side folder configurations * Trading partner code identification for partner document management and archiving ## Overview Infor connectors perform two functions when sending or receiving EDI documents to Infor CloudSuite Industrial. When sending, the connector converts XML documents into a flat file structure accepted by CloudSuite, then sends the resulting file to the remote data folder where it can be picked up by Infor. When receiving, the connector polls the data folder for flat files, which it then converts into XML. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Configuration *Settings related to the core configuration of the connector.*
Setting Description
**Connector Id** The static, unique identifier for the connector.
**Connector Type** Displays the connector name and a description of what it does.
**Connector Description** An optional field to provide a free-form description of the connector and its role in the flow.
#### Infor System Configuration *Settings provided in the connection to the Infor system.* | Setting | Description | | - | - | | **Trading Partner Code** | The code identifying the documents in the Infor system. | #### Inbound *Folder locations for demand-side documents in the Infor system.* | Setting | Description | | - | - | | **Inbound Data Logic Folder** | The folder location for demand-side file pickup and drop-off. | | **Inbound Archive Folder** | The folder location for archiving processed demand-side documents. | #### Outbound *Folder locations for outbound documents in the Infor system.* | Setting | Description | | - | - | | **Outbound Data Logic Folder** | The folder location for supply-side file pickup and drop-off. | | **Outbound Archive Folder** | The folder location for archiving processed supply-side documents. | #### Other Settings | Setting | Description | | - | - | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | ### Advanced Tab #### Other Settings | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | #### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection The following settings are required to connect to Infor: * **Trading Partner Code** identifies the documents in the Infor system. * **Inbound Data Logic Folder** is for demand-side file pickup and drop-off. * **Inbound Archive Folder** is for archiving processed demand-side documents. * **Outbound Data Logic Folder** is for supply-side file pickup and drop-off. * **Outbound Archive Folder** is for archiving processed supply-side documents. After these fields have been specified, the connector can exchange documents with the Infor system. ## Sending Files After a connection to Infor has been established, XML files in the **Transactions** tab of the Infor connector are converted to the Infor standard flat file format and uploaded. If **Send Automation** is enabled on the [Automation tab](#automation-tab), the connector automatically polls this folder for processing. ## Receiving Files Files that are found when polling Infor CloudSuite are downloaded to the Infor connector **Transactions** tab. The standard flat file structure is automatically converted into XML. If the Infor connector is connected to other connectors in a flow, the file is passed to the next connector. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# Kintone Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone Connect Arc to Kintone for data integration and synchronization The Kintone connector allows you to integrate Kintone into your data flow by pushing or pulling data from Kintone. ## Key Capabilities * Cloud-based business application platform integration for custom app data management * Flexible authentication options including Password, API Token or OAuth * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Kintone. ## Establish a Connection To establish a connection to Kintone, use one of the following two methods: * Add a Kintone connector to your flow. Then, on the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Kintone**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Kintone can be reused across multiple Kintone connectors. ## Enter Connection Settings 1. Enter the connection information: * **Name**: The static name of the connection. Set this as desired. * **Type**: This is always set to Kintone. * **Connection**: The authentication scheme to use for the connection. Options are **Password**, **API Token**, and **OAuth**. * **URL**: The URL to the Kintone site you are connecting to. 2. Enter the connection information for your chosen authentication scheme: * [Password](#password) * [API Token](#api-token) * [OAuth](#oauth) 3. Select your newly-created connection in the **Connection** drop-down list of the connector configuration pane. ### Password 1. In the **User** field, enter the username to use for logging in. 2. In the **Password** field, enter the password for the user. 3. If needed, click **Advanced** to open the menu of advanced connection settings. These should not be needed in most cases. 4. Click **Test Connection** to ensure that Arc can connect to Kintone with the provided information. If an error occurs, check all fields and try again. 5. Click **Add Connection** to finalize the connection. ### API Token 1. In the **ApiToken** field, provide the API token to use for authentication. 2. If needed, click **Advanced** to open the menu of advanced connection settings. These should not be needed in most cases. 3. Click **Test Connection** to ensure that Arc can connect to Kintone with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. ### OAuth 1. In the **OAuth Client Id** field, enter the client Id for your account. 2. In the **OAuth Client Secret** field, enter the client secret for your account. 3. If needed, click **Advanced** to open the menu of advanced connection settings. These should not be needed in most cases. 4. Click **Connect**. 5. On the new tab that opens, log into your Kintone account. If prompted, provide access permissions. 6. Return to Arc. * If the connection succeeds, click **Add Connection**. * If the connection fails, check your URL and login credentials and try again. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Kintone, you must choose the action that the Kintone connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./kintone-upsert) | Inserts or updates Kintone data. By default, if a record already exists in Kintone, an update is performed on the existing data in Kintone using the values provided from the input. | End | | [Lookup](./kintone-lookup) | Retrieves values from Kintone and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Kintone. It should be formatted as a SQL query against the Kintone tables. | Middle | | [Lookup Stored Procedure](./kintone-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./kintone-select) | Retrieves data from Kintone and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./kintone-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./kintone-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | | Table | Evaluates to the name of the table where the connector is selecting data from. | | PK | Evaluates to the primary key value of the received record from the table. | | Sequence | Evaluates to a four digit number based on how many total records were received by the connector (for example, %Sequence% evaluates to 0005 for the fifth record returned). | | FileFormat | Evaluates to the output file format specified on the connector's Advanced tab (XML, CSV, or TSV). | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Kintone Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-ac Advanced configuration options for the Kintone connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Kintone connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Kintone connector models Kintone tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Kintone values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Kintone connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Kintone. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Kintone connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Kintone connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Kintone. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Kintone, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Kintone. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Kintone connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Kintone Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Kintone connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Kintone connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Kintone Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-lookup Retrieve values from Kintone and insert them into an Arc message in the flow. The Lookup action retrieves values from Kintone and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Kintone connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Kintone connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Kintone connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Kintone connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Kintone tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Kintone connector can perform Lookup queries against a local cache instead of querying Kintone directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Kintone on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Kintone directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Kintone.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Kintone Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-lsp Use a stored procedure to retrieve values from Kintone and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Kintone connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Kintone connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Kintone connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Kintone Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-select Retrieve data from Kintone and bring it into Arc. The Select action retrieves data from Kintone and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Kintone. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Kintone, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Kintone. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/kintone/kintone-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Kintone relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Kintone supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Kintone connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Kintone Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Kintone\_Select** connector pulls data from a Kintone connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Kintone\_Select** then passes down the flow from **CSV1**.
# Kintone Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/kintone/kintone-upsert Insert or update Kintone data from an Arc message in the flow. The Upsert action updates or inserts Kintone data. By default, if a record already exists in Kintone, an update is performed on the existing data in Kintone using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Kintone Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Kintone. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/kintone/kintone-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Kintone connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Kintone Upsert connector at the end: ![](/26.4/images/kintone_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Kintone\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Kintone.
# MQTT Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/mqtt Connect Arc to an MQTT broker to publish messages and subscribe to topics using the Message Queuing Telemetry Transport protocol. The OASIS Message Queuing Telemetry Transport (MQTT) connector supports publishing MQTT messages and subscribing to MQTT topics. ## Key Capabilities * Bidirectional IoT message queuing with multiple QoS levels * TLS encryption and client certificate authentication with configurable keep-alive intervals * Topic-based filtering with wildcard support and Last Will message handling ## Overview Sending data through an MQTT connector publishes the data to the configured topic, and messages that are received on any of the topics that the connector is subscribed to are placed in the **Transactions** tab. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to the remote MQTT server connection.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Connection** | A list of available connections. Select an existing connection or click **Create** to add a new one. | | **Action** | Whether to [Publish](#publishing) messages to a topic on an MQTT broker or [Subscribe](#subscribing) to receive messages from one or more MQTT topic filters. | | **Server Status** | (Only applicable when **Action** is **Subscribe**) The state of the connector's server: **Active** (the server is running; click **Stop Server** to stop it), **Inactive** (the server is not running; click **Start Server** to start it). To review the logs the server produces while it runs, see the [Logs tab](#logs-tab). | #### Client Authentication *Settings related to authenticating against the remote MQTT server.* | Setting | Description | | - | - | | **Username** | The username credential with which to authenticate. | | **Password** | The password for the specified username. | #### TLS Settings *Settings related to establishing a secure connection via TLS.* | Setting | Description | | - | - | | **Use TLS** | Whether the connector should negotiate TLS when connecting to the remote server. | | **TLS Server Certificate** | The public certificate used to verify the identity of the remote server when establishing a TLS connection. Set this to **Any Certificate** to implicitly trust the target server. | #### Publish *Settings related to publishing MQTT messages.* | Setting | Description | | - | - | | **Topic** | The topic for the message to publish. | | **QoS** | The service level for message delivery. Options are: *at most once*, *at least once*, or *exactly once*. See [Publishing](#publishing) for more information. | #### Subscribe *Settings related to subscribing to MQTT messages.* | Setting | Description | | - | - | | **Enable MQTT Receiver** | Whether the connector should receive MQTT messages. | | **Topic Filters** | A comma-separated list of topics for the connector to subscribe to. | | **QoS** | The service level for message delivery. Options are: *at most once*, *at least once*, or *exactly once*. See [Publishing](#publishing) for more information. | ### Advanced Tab #### TLS Client Authentication *Settings related to client authentication when two-way TLS authentication is required.*
SettingDescription
**Private Certificate** The private certificate presented during TLS client authentication.
**Certificate Password** The password required to access the TLS client certificate.
#### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Keep Alive Interval** | The maximum period of inactivity the connector allows before sending a keep-alive packet. | | **Last Will Message** | The message that the server should publish in the event of an ungraceful disconnection. | | **Last Will Topic** | The topic of the Last Will Message. | | **Republish Interval** | The number of seconds that the connector waits before republishing unacknowledged messages. | | **Timeout** | The length of time (in seconds) the connector waits for a connection response before throwing a timeout error. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent as messages to the MQTT server. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ### Logs Tab The **Logs** tab lists the service logs generated by this connector. Each row represents a log file and shows its date modified, file name, size, and status. Use the search bar or the status filter to find a specific log, click a file name to preview or download it, or click **Download All Logs** to download all of them as a single zip file. ## Establishing a Connection At a minimum, set **Remote Host** on the [Settings tab](#settings-tab) to the domain name or IP address of the MQTT server. If you are using a non-standard port, set **Remote Port**. If authentication is required, set **Username** and **Password**. If you require a secure connection, check **Use TLS**, and optionally set the **TLS Server Certificate** to the server's public certificate, or choose *Any Certificate* to accept any certificate presented by the server. If this field is used improperly, it creates a security risk. Use extreme caution when setting this field. ## Publishing To publish data, set **Topic** in the [Publish](#publish) section to the topic where the connector should publish messages. **QoS** values range from 0 to 2 and have the following meanings: * 0 — At most once. The published message is sent once, and if it does not arrive it is lost. * 1 — At least once. Guarantees that the published message arrives, but there might be duplicates. * 2 — Exactly once. Guarantees that the published message arrives and that there are no duplicates. In the [Advanced Settings](#advanced-settings) portion of the Advanced tab, you can set **Republish Interval** to the number of seconds that you want the connector to wait for acknowledgements to messages that require them. If this time elapses without receiving an acknowledgement, the connector republishes the message. Republished messages reuse the packet Id from the original message, and automatically set the Duplicate flag on the message. Specifying a value of 0 for **Republish Interval** prevents the connector from automatically republishing messages. ## Subscribing In the [Subscribe](#subscribe) section of the settings page, you can check **Enable MQTT Receiver** to indicate that the connector should also receive data. When checked, the connector subscribes to the topics you specify in **Topic Filters**, and incoming messages are automatically placed on the **Transactions** tab, named based on the value specified in **Local File Scheme** in the [Advanced Settings](#advanced-settings) portion of the Advanced tab. By default, the connector names files with the topic and packet Id of the message, using `%topic%_%packetid%.data` as the **Local File Scheme** value. You can supply a comma-separated list of topic filters that the connector should subscribe to in **Topic Filters**. The following characters have special meanings in a topic filter: * / — The topic level separator * \# — The multi-level wildcard (zero or more levels) * \+ — The single-level wildcard (exactly one level) * Leading \$ — Denotes a "system topic" You can use **Last Will Message** and **Last Will Topic** on the Advanced tab to specify the message that the server should publish, and the topic it should publish that message to, in the event of an ungraceful disconnection. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# NetSuite Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite Connect Arc to NetSuite for data integration and synchronization The NetSuite connector allows you to integrate NetSuite into your data flow by pushing or pulling data from NetSuite. ## Key Capabilities * Complete ERP integration with OAuth, Token, and Basic authentication options * Account-specific connectivity with NetSuite Account ID configuration * Full access to NetSuite business objects (customers, orders, items, financials) * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to NetSuite. ## Establish a Connection To allow Arc to use data from NetSuite, you must first establish a connection to NetSuite. There are two ways to establish this connection: * Add a NetSuite connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **NetSuite**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to NetSuite can be reused across multiple NetSuite connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to NetSuite. * **Auth Scheme** — The authentication scheme to use when connecting to NetSuite. Options are **Basic**, **Token**, and **OAuth**. * **AccountId** — The NetSuite Account Id. * **User / Password** — (Basic Auth Scheme only) The user and password to use to log in to NetSuite. * **OAuthClientId / OAuthClientSecret** — (Token and OAuth Auth Schemes only) The OAuth Client ID and Client Secret for logging in. * **AccessToken / AccessTokenSecret** — (Token Auth Scheme only) The Access Token and Secret for logging in. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to NetSuite with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Custom Schemas Tab The **Custom Schemas** tab appears in the [connection settings](/26.4/self-hosted/en/getting-started/administration/settings/connections) flyout for connectors whose underlying driver supports schema-creation stored procedures. For NetSuite, this includes stored procedures such as `CreateSavedSearchSchema`, `CreateRestletSchema`, and `CreateSchema`, which let you define additional tables (such as SavedSearches or RESTlet-based endpoints) that are not discoverable through the standard driver metadata. Custom schemas are stored at the connection level, so all NetSuite connectors sharing the same connection automatically have access to them. You do not need to configure a file path or manage schema files on disk, and custom schemas are preserved when you import or export your Arc configuration. ### Add a Custom Schema > [!NOTE] > > Before you begin, you must have successfully created and tested the connection. The **Custom Schemas** tab does not appear until that step is complete. 1. Open the connection's settings and go to the **Custom Schemas** tab. 2. Click **Add Custom Schema**. 3. In the **Select Stored Procedure** pane, select the procedure that matches the type of schema you want to create (for example, **CreateSavedSearchSchema** for a NetSuite SavedSearch). Use the search box to filter by name. ![Select Stored Procedure pane](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/custom_schema_select_sp.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=6e8a9aee2d54f09c0436bd6d7e915b3a) 4. Fill in the required fields. The fields vary depending on the stored procedure you selected. See [Stored Procedure Parameters](#stored-procedure-parameters) for details. The following image shows the **CreateSchema** stored procedure. ![Custom schema details form](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/custom_schema_details.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1a1c6c191c28771dd6f51c6f22bacc29) 5. Click **Add Schema**. The new schema appears in the list and is immediately available to all connectors using this connection. #### Stored Procedure Parameters The available stored procedure parameters depend on which stored procedure you selected in step 3. Required fields are marked with an asterisk (\*). **CreateSavedSearchSchema** | Field | Description | | - | - | | **SavedSearch** \* | The name of the saved search. This becomes the name of the resulting table. | | **SavedSearchId** \* | The InternalId of the saved search. | | **SavedSearchDescription** | An optional description of the saved search. | | **SearchType** \* | The type of search being used. Select a value from the list. | | **ResultsToSample** | An integer designating the number of results to sample. | **CreateRestletSchema** | Field | Description | | - | - | | **TableName** \* | The name of the RESTlet schema to create. | | **Description** | A description of the table schema to save. | | **SearchId** \* | The Id of the search to use, as displayed in the NetSuite UI (for example, customsearch123). | | **ScriptId** \* | The ScriptId of the RESTlet to create a schema for. See [Upload the Search Script for RESTlet Schemas](#upload-the-search-script-for-restlet-schemas). | | **DeploymentNum** \* | The deployment number of the RESTlet schema to execute. Typically this is 1 unless a different deployment is required. | | **UseLabels** | A boolean indicating if labels should be used for column names. | | **FlattenSelects** | A boolean indicating if select response types should be flattened if possible. NetSuite returns structured data as a select including record references. Set this to false if multiselect columns must be retrieved, as those are not supported by this option. | **CreateSchema** | Field | Description | | - | - | | **TableName** \* | The name of the table or view for which to generate the schema. Provide the name exactly as it appears in NetSuite. | #### Upload the Search Script for RESTlet Schemas Before you create a schema with `CreateRestletSchema`, you must create a RESTlet in NetSuite that runs your saved search. The RESTlet depends on a SuiteScript file, `search_script.js`, that CData provides. 1. In NetSuite, go to **Setup > Company > Enable Features**. On the **SuiteCloud** tab, enable **Client SuiteScript**, **Server SuiteScript**, and **SuiteScript Server Pages**. 2. [Download search\_script.js](https://downloads.cdata.com/cdatabuilds/support/apps/arc/netsuite/search_script.js). 3. In NetSuite, upload the file to the File Cabinet (**Documents > Files > SuiteScripts**). 4. Still in NetSuite, create a script record from the uploaded file, and then create a deployment for it: 1. Go to **Customization > Scripting > Scripts > New**, select `search_script.js`, and click **Create Script Record**. 2. Enter a script name, add a title on the **Deployments** tab, and click **Save**. 3. Set the deployment status to **Released** and add the roles that will use the connection on the **Audience** tab. A deployment with a status of **Testing** returns saved search results only to the user who created it. 4. Note the script Id and the deployment number, which appear in the deployment URL. You enter these values as **ScriptId** and **DeploymentNum** in the next step. 5. In Arc, return to [Add a Custom Schema](#add-a-custom-schema) and create the schema with the `CreateRestletSchema` stored procedure. For the full NetSuite procedure for creating RESTlets, see the [CData NetSuite driver documentation](https://cdn.cdata.com/help/DNN/ado/pg_restlets.htm). Ignore that procedure's instruction to locate `search_script.js` in the driver installation directory. Use the download link above instead. ### Delete a Custom Schema On the **Custom Schemas** tab, click the delete icon next to the schema you want to remove. The schema is deleted and the list refreshes automatically. ## Select an Action After establishing a connection to NetSuite, you must choose the action that the NetSuite connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./netsuite-upsert) | Inserts or updates NetSuite data. By default, if a record already exists in NetSuite, an update is performed on the existing data in NetSuite using the values provided from the input. | End | | [Lookup](./netsuite-lookup) | Retrieves values from NetSuite and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from NetSuite. It should be formatted as a SQL query against the NetSuite tables. | Middle | | [Lookup Stored Procedure](./netsuite-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./netsuite-select) | Retrieves data from NetSuite and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./netsuite-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./netsuite-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # NetSuite Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-ac Advanced configuration options for the NetSuite connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the NetSuite connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The NetSuite connector models NetSuite tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and NetSuite values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The NetSuite connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into NetSuite. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The NetSuite connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the NetSuite connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by NetSuite. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by NetSuite, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from NetSuite. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure NetSuite connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # NetSuite Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the NetSuite connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the NetSuite connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # NetSuite Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-lookup Retrieve values from NetSuite and insert them into an Arc message in the flow. The Lookup action retrieves values from NetSuite and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the NetSuite connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the NetSuite connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the NetSuite connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the NetSuite connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from NetSuite tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The NetSuite connector can perform Lookup queries against a local cache instead of querying NetSuite directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query NetSuite on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries NetSuite directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to NetSuite.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # NetSuite Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-lsp Use a stored procedure to retrieve values from NetSuite and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the NetSuite connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the NetSuite connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the NetSuite connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# NetSuite Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-select Retrieve data from NetSuite and bring it into Arc. The Select action retrieves data from NetSuite and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries NetSuite. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to NetSuite, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from NetSuite. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/netsuite/netsuite-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in NetSuite relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what NetSuite supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured NetSuite connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a NetSuite Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **NetSuite\_Select** connector pulls data from a NetSuite connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **NetSuite\_Select** then passes down the flow from **CSV1**.
# NetSuite Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/netsuite/netsuite-upsert Insert or update NetSuite data from an Arc message in the flow. The Upsert action updates or inserts NetSuite data. By default, if a record already exists in NetSuite, an update is performed on the existing data in NetSuite using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The NetSuite Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from NetSuite. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/netsuite/netsuite-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured NetSuite connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a NetSuite Upsert connector at the end: ![](/26.4/images/netsuite_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **NetSuite\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to NetSuite.
# Odoo Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo Connect Arc to Odoo for data integration and synchronization The Odoo connector allows you to integrate Odoo into your data flow by pushing or pulling data from Odoo. ## Key Capabilities * Open-source ERP and CRM platform integration with multi-module support across sales, inventory, accounting, and operations * API-based connectivity * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Odoo. ## Establish a Connection To allow Arc to use data from Odoo, you must first establish a connection to Odoo. There are two ways to establish this connection: * Add a Odoo connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Odoo**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Odoo can be reused across multiple Odoo connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Odoo. * **URL** — The URL to a valid Odoo site. * **User** — The user with which you want to connect. * **APIToken / Password** — The API token for the user with which you want to connect. **Note:** API tokens are supported only in Odoo 14 and later. Therefore, if your version of Odoo is earlier that Odoo 14, you can provide a password in the **APIToken** field. * **Database** — The Odoo database * **Odoo Edition** — The edition of Odoo you are using. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Odoo with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Odoo, you must choose the action that the Odoo connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./odoo-upsert) | Inserts or updates Odoo data. By default, if a record already exists in Odoo, an update is performed on the existing data in Odoo using the values provided from the input. | End | | [Lookup](./odoo-lookup) | Retrieves values from Odoo and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Odoo. It should be formatted as a SQL query against the Odoo tables. | Middle | | [Lookup Stored Procedure](./odoo-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./odoo-select) | Retrieves data from Odoo and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./odoo-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./odoo-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Odoo Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-ac Advanced configuration options for the Odoo connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Odoo connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Odoo connector models Odoo tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Odoo values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Odoo connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Odoo. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Odoo connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Odoo connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Odoo. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Odoo, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Odoo. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Odoo connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Odoo Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Odoo connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Odoo connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Odoo Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-lookup Retrieve values from Odoo and insert them into an Arc message in the flow. The Lookup action retrieves values from Odoo and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Odoo connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Odoo connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Odoo connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Odoo connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Odoo tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Odoo connector can perform Lookup queries against a local cache instead of querying Odoo directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Odoo on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Odoo directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Odoo.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Odoo Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-lsp Use a stored procedure to retrieve values from Odoo and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Odoo connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Odoo connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Odoo connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Odoo Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-select Retrieve data from Odoo and bring it into Arc. The Select action retrieves data from Odoo and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Odoo. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Odoo, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Odoo. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/odoo/odoo-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Odoo relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Odoo supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Odoo connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Odoo Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Odoo\_Select** connector pulls data from a Odoo connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Odoo\_Select** then passes down the flow from **CSV1**.
# Odoo Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/odoo/odoo-upsert Insert or update Odoo data from an Arc message in the flow. The Upsert action updates or inserts Odoo data. By default, if a record already exists in Odoo, an update is performed on the existing data in Odoo using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Odoo Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Odoo. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/odoo/odoo-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Odoo connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Odoo Upsert connector at the end: ![](/26.4/images/odoo_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Odoo\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Odoo.
# QuickBooks Online Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online Connect Arc to QuickBooks Online for data integration and synchronization The QuickBooks Online connector allows you to integrate QuickBooks Online into your data flow by pushing or pulling data from QuickBooks Online. ## Key Capabilities * Cloud-based accounting system integration with OAuth 2.0 authentication * Direct access to financial data including invoices, payments, and chart of accounts * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to QuickBooks Online. ## Establish a Connection To allow Arc to use data from QuickBooks Online, you must first establish a connection to QuickBooks Online. There are two ways to establish this connection: * Add a QuickBooks Online connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **QuickBooks Online**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to QuickBooks Online can be reused across multiple QuickBooks Online connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter a name for the connection. 2. Click **Connect**. 3. A new window opens for logging into QuickBooks Online. Enter your credentials and log in. 4. If prompted, grant permissions for Arc to connect to QuickBooks Online. You then return to Arc. 5. Click **Add Connection** to finalize the connection. 6. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 7. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to QuickBooks Online, you must choose the action that the QuickBooks Online connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./quickbooks-online-upsert) | Inserts or updates QuickBooks Online data. By default, if a record already exists in QuickBooks Online, an update is performed on the existing data in QuickBooks Online using the values provided from the input. | End | | [Lookup](./quickbooks-online-lookup) | Retrieves values from QuickBooks Online and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from QuickBooks Online. It should be formatted as a SQL query against the QuickBooks Online tables. | Middle | | [Lookup Stored Procedure](./quickbooks-online-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./quickbooks-online-select) | Retrieves data from QuickBooks Online and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./quickbooks-online-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./quickbooks-online-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # QuickBooks Online Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-ac Advanced configuration options for the QuickBooks Online connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the QuickBooks Online connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The QuickBooks Online connector models QuickBooks Online tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and QuickBooks Online values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The QuickBooks Online connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into QuickBooks Online. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The QuickBooks Online connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the QuickBooks Online connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by QuickBooks Online. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by QuickBooks Online, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from QuickBooks Online. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure QuickBooks Online connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # QuickBooks Online Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the QuickBooks Online connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the QuickBooks Online connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # QuickBooks Online Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-lookup Retrieve values from QuickBooks Online and insert them into an Arc message in the flow. The Lookup action retrieves values from QuickBooks Online and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the QuickBooks Online connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the QuickBooks Online connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the QuickBooks Online connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the QuickBooks Online connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from QuickBooks Online tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The QuickBooks Online connector can perform Lookup queries against a local cache instead of querying QuickBooks Online directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query QuickBooks Online on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries QuickBooks Online directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to QuickBooks Online.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # QuickBooks Online Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-lsp Use a stored procedure to retrieve values from QuickBooks Online and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the QuickBooks Online connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the QuickBooks Online connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the QuickBooks Online connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# QuickBooks Online Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-select Retrieve data from QuickBooks Online and bring it into Arc. The Select action retrieves data from QuickBooks Online and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries QuickBooks Online. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to QuickBooks Online, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from QuickBooks Online. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in QuickBooks Online relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what QuickBooks Online supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured QuickBooks Online connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a QuickBooks Online Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **QuickBooks Online\_Select** connector pulls data from a QuickBooks Online connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **QuickBooks Online\_Select** then passes down the flow from **CSV1**.
# QuickBooks Online Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-upsert Insert or update QuickBooks Online data from an Arc message in the flow. The Upsert action updates or inserts QuickBooks Online data. By default, if a record already exists in QuickBooks Online, an update is performed on the existing data in QuickBooks Online using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The QuickBooks Online Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from QuickBooks Online. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/quickbooks-online/quickbooks-online-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured QuickBooks Online connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a QuickBooks Online Upsert connector at the end: ![](/26.4/images/qbo_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **QuickBooks Online\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to QuickBooks Online.
# Sage Intacct Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct Connect Arc to Sage Intacct for data integration and synchronization The Sage Intacct connector allows you to integrate Sage Intacct into your data flow by pushing or pulling data from Sage Intacct. ## Key Capabilities * Cloud-based accounting software integration for general ledger, accounts payable/receivable, and financial reporting * Basic and API Token authentication options with real-time financial data transfers * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Sage Intacct. ## Establish a Connection To allow Arc to use data from Sage Intacct, you must first establish a connection to Sage Intacct. There are two ways to establish this connection: * Add a Sage Intacct connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Sage Intacct**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Sage Intacct can be reused across multiple Sage Intacct connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Sage Intacct. * **Auth Scheme** — The authentication scheme to use for the connection. Options are **Basic** and **Okta**. * **Company Id** — (Basic auth scheme only) The Company Id for authentication. * **SSO Login URL** — (Okta auth scheme only) The single sign-on (SSO) URL for authentication. * **User** — The username to use for logging in. * **Password** — The password for the user entered above. * **Intacct User Id** — (Okta auth scheme only) The User Id for authentication. * **API Token** — (Okta auth scheme only) The API token for authentication. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Sage Intacct with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Sage Intacct, you must choose the action that the Sage Intacct connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./sage-intacct-upsert) | Inserts or updates Sage Intacct data. By default, if a record already exists in Sage Intacct, an update is performed on the existing data in Sage Intacct using the values provided from the input. | End | | [Lookup](./sage-intacct-lookup) | Retrieves values from Sage Intacct and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Sage Intacct. It should be formatted as a SQL query against the Sage Intacct tables. | Middle | | [Lookup Stored Procedure](./sage-intacct-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./sage-intacct-select) | Retrieves data from Sage Intacct and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./sage-intacct-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./sage-intacct-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Sage Intacct Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-ac Advanced configuration options for the Sage Intacct connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Sage Intacct connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Sage Intacct connector models Sage Intacct tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Sage Intacct values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Sage Intacct connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Sage Intacct. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Sage Intacct connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Sage Intacct connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Sage Intacct. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Sage Intacct, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Sage Intacct. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Sage Intacct connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Sage Intacct Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Sage Intacct connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Sage Intacct connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Sage Intacct Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-lookup Retrieve values from Sage Intacct and insert them into an Arc message in the flow. The Lookup action retrieves values from Sage Intacct and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Sage Intacct connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Sage Intacct connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Sage Intacct connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Sage Intacct connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Sage Intacct tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Sage Intacct connector can perform Lookup queries against a local cache instead of querying Sage Intacct directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Sage Intacct on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Sage Intacct directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Sage Intacct.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Sage Intacct Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-lsp Use a stored procedure to retrieve values from Sage Intacct and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Sage Intacct connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Sage Intacct connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Sage Intacct connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Sage Intacct Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-select Retrieve data from Sage Intacct and bring it into Arc. The Select action retrieves data from Sage Intacct and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Sage Intacct. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Sage Intacct, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Sage Intacct. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Sage Intacct relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Sage Intacct supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Sage Intacct connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Sage Intacct Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Sage Intacct\_Select** connector pulls data from a Sage Intacct connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Sage Intacct\_Select** then passes down the flow from **CSV1**.
# Sage Intacct Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-upsert Insert or update Sage Intacct data from an Arc message in the flow. The Upsert action updates or inserts Sage Intacct data. By default, if a record already exists in Sage Intacct, an update is performed on the existing data in Sage Intacct using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Sage Intacct Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Sage Intacct. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/sage-intacct/sage-intacct-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Sage Intacct connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Sage Intacct Upsert connector at the end: ![](/26.4/images/sage_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Sage Intacct\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Sage Intacct.
# Salesforce Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce Connect Arc to Salesforce for data integration and synchronization The Salesforce connector allows you to integrate Salesforce into your data flow by pushing or pulling data from Salesforce. ## Key Capabilities * Leading CRM platform integration with OAuth authentication and sandbox support * Specialized document upload/download capabilities for Salesforce Files and Content * Complete access to Salesforce objects (leads, contacts, accounts, opportunities) * Standard database connector actions (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Salesforce. ## Establish a Connection To allow Arc to use data from Salesforce, you must first establish a connection to Salesforce. There are two ways to establish this connection: * Add a Salesforce connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Salesforce**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Salesforce can be reused across multiple Salesforce connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Enter a name for the connection. 2. Set **Use Sandbox** to **True** to create a connection to your sandbox account. 3. Click **Connect**. 4. A new window opens for logging into Salesforce. Enter your credentials and log in. 5. If prompted, grant permissions for Arc to connect to Salesforce. You then return to Arc. 6. Click **Add Connection** to finalize the connection. 7. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 8. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Salesforce, you must choose the action that the Salesforce connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./salesforce-upsert) | Inserts or updates Salesforce data. By default, if a record already exists in Salesforce, an update is performed on the existing data in Salesforce using the values provided from the input. | End | | [Lookup](./salesforce-lookup) | Retrieves values from Salesforce and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Salesforce. It should be formatted as a SQL query against the Salesforce tables. | Middle | | [Lookup Stored Procedure](./salesforce-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./salesforce-select) | Retrieves data from Salesforce and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./salesforce-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Upload](./salesforce-upload) | Uploads documents from the flow into Salesforce. | End | | [Download](./salesforce-download) | Downloads documents from Salesforce and passes them down the flow. | Beginning | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./salesforce-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Salesforce Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-ac Reference for advanced settings available on the Salesforce connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the Salesforce connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Salesforce connector models Salesforce tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Salesforce values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Salesforce connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Salesforce. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Salesforce connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Salesforce connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Salesforce. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Salesforce, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Salesforce. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Salesforce connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Salesforce Download Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-download Download documents from Salesforce and pass them down the flow. The Download action pulls data from Salesforce and passes it down the flow. The Download connector belongs at the beginning of an Arc [flow](../../flows/flows). In the connector configuration pane, enter a **File Mask** and a **Folder Name**. Files that match those criteria are then downloaded when the connector attempts to receive files. ![Salesforce Download configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/salesforce_download.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=079a38ae47721c4f729e6f4e05b0a283) # Salesforce Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-esp Execute a Salesforce stored procedure and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Salesforce connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Salesforce connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Salesforce Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-lookup Retrieve values from Salesforce and insert them into an Arc message in the flow. The Lookup action retrieves values from Salesforce and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Salesforce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Salesforce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Salesforce connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Salesforce connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Salesforce tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Salesforce connector can perform Lookup queries against a local cache instead of querying Salesforce directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Salesforce on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Salesforce directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Salesforce.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Salesforce Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-lsp Use a Salesforce stored procedure to look up values and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Salesforce connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Salesforce connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Salesforce connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Salesforce Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-select Retrieve data from Salesforce and bring it into Arc at the beginning of a flow. The Select action retrieves data from Salesforce and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Salesforce. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Salesforce, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Salesforce. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/salesforce/salesforce-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Salesforce relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Salesforce supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Salesforce connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Salesforce Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Salesforce\_Select** connector pulls data from a Salesforce connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Salesforce\_Select** then passes down the flow from **CSV1**.
# Salesforce Upload Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-upload Upload documents from the flow into Salesforce. The Upload action pushes data from the flow into Salesforce. The Upload connector belongs at the end of an Arc [flow](../../flows/flows). In the connector configuration pane, specify a **Folder Name**. Files that flow into the Salesforce Upload connector are then uploaded into Salesforce in the specified folder. ![Salesforce Upload configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/salesforce_upload.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=de33326979b536ccf4e7cf1f0740f394) # Salesforce Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/salesforce/salesforce-upsert Insert or update records in Salesforce from an Arc message in the flow. The Upsert action updates or inserts Salesforce data. By default, if a record already exists in Salesforce, an update is performed on the existing data in Salesforce using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Salesforce Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Salesforce. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/salesforce/salesforce-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Salesforce connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Salesforce Upsert connector at the end: ![](/26.4/images/salesforce_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Salesforce\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Salesforce.
# SAP IDoc Connector Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sap-idoc/sap-idoc Bidirectional SAP IDoc data exchange supporting NetWeaver RFC, Classic RFC, JCO, and SOAP connection methods. An SAP IDoc is a data container that is used to exchange information between SAP and another process that can understand the syntax and semantics of the data. The connector supports sending and receiving IDoc files to and from a variety of SAP systems. You can also use it to translate IDocs without needing to connect to SAP. ## Key Capabilities * Comprehensive SAP IDoc integration supporting multiple connection methods (NetWeaver RFC SDK, Classic RFC SDK, JCO, and SOAP) * Bidirectional data exchange with support for raw IDoc and IDoc XML formats * Automatic schema refresh and validation with control record header management for multi-SAP-system routing ## Overview IDoc connectors can connect to SAP using the following methods: **Windows** * NetWeaver RFC SDK * Classic RFC SDK * Classic RFC SDK with Unicode support * SOAP (send-only) **Java** * Java connector (JCO) * SOAP (send-only) IDoc connectors can send, receive, and translate files formatted either as raw IDoc or as IDoc XML. ## Prerequisites If you are using the SAP IDoc connector in [Send](./sap-idoc-send) or [Receive](./sap-idoc-receive) mode, you must have a live SAP IDoc system and a clear network path to that system from Arc. You also need to install the necessary dependencies on your machine. Follow the instructions below for your connection type. > [!NOTE] > > If you are using the connector in [Translate](./sap-idoc-translate) mode, these prerequisites are not required. ### Using NetWeaver RFC SDK The following libraries from the NetWeaver RFC SDK must be placed into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems): * sapnwrfc.dll * icudt30.dll * icuin30.dll * icuuc30.dll * libicudecnumber.dll * libsapucum.dll > [!NOTE] > > * The .dll names might be different based on the version of the SAP RFC SDK you have installed. The examples above are for version 3.0. > * You need to add the `sapgw00` service to your listed services for the NetWeaver SDK connection to work on your client machine. You can add it to the services file at `C:\\Windows\\System32\\drivers\\etc\\services`. ### Using Classic RFC SDK The non-Unicode library does not require anything other than librfc32.dll. Place librfc32.dll into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems). > [!NOTE] > > These libraries are provided in both x86 and x64, so you must use the same platform that the process is using. If the application is running on a 64-bit version of Windows, the process is 64-bit. ### Using Classic RFC SDK with Unicode Support The following libraries from the SAP RFC SDK must be placed into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems): * librfc32u.dll * icudt30.dll * icuin30.dll * icuuc30.dll > [!NOTE] > > * The .dll names might be different based on the version of the SAP RFC SDK you have installed. The examples above are for version 3.0. > * These libraries are provided in both x86 and x64, so you must use the same platform that the process is using. If the application is running on a 64-bit version of Windows, the process is 64-bit. ### Using the Java Connector To use the Java connector, add sapjco3.jar to the `lib` folder of your Java servlet container. ### Using SOAP SOAP can only be used to [send](./sap-idoc-send) IDocs to SAP. You must use a different connection type to [receive](./sap-idoc-receive) IDocs from SAP. > [!NOTE] > > This requires enabling SOAP connections on the SAP system. ## Add a Connection To allow Arc to send or receive SAP data, you must first establish a connection to SAP. You have two ways to establish this connection: * Add a SAP IDoc connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **SAP IDoc**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to SAP IDoc can be re-used across multiple SAP IDoc connectors. ### Enter Connection Settings After opening a new connection dialogue, give it a **Name**, then complete the other options. #### Host Configuration *Settings related to establishing the connection to the SAP system.* | Setting | Description | | - | - | | **Connect Type** | The method with which to connect to SAP. See the [Establishing a Connection](#establishing-a-connection) section for your method for additional information. | | **SAP Host** | Hostname or IP address of the target SAP system. | | **System Number** | The number by which the target SAP system is defined. Valid values are 0-99, and often this value should be 0. | #### Client Authentication *Settings related to authenticating against the SAP system.* | Setting | Description | | - | - | | **Client** | The client that is authenticating to the SAP system. | | **User** | The username credential for authentication. | | **Password** | The password for the specified user. | #### Language Settings *Settings related to communication language.* | Setting | Description | | - | - | | **Language** | The language value to be used when connecting to the SAP system. Set this to the ISO 639-1 value for the language the SAP system uses. | #### Gateway *Settings related to communicating with SAP through a gateway.* | Setting | Description | | - | - | | **Gateway Host** | The hostname or IP address of the gateway through which to connect. | | **Gateway Service** | The gateway service to use. | #### Logon Load Balancing *Settings related to logging in to the SAP system as a group to enable load balancing.* | Setting | Description | | - | - | | **Message Server** | The service name of the load balancer. | | **System ID** | Identifies the SAP system. | | **Logon Group** | The group to use when load balancing. | #### Secure Network Communications (SNC) *Settings related to using an external security product to enhance the SAP connection security.* | Setting | Description | | - | - | | **SNC Name** | The SNC name for the machine making the connection. | | **SNC Partner Name** | The SNC name of the SAP system the connector is connecting to. | | **SNC Quality of Protection Level** | The level of protection added by the SNC product. | | **SNC Library** | The path to the external security product's library. | Click **Test Connection** to ensure that Arc can connect to SAP with the provided information. If an error occurs, check all fields and try again. Click **Add Connection** to save your changes. > [!TIP] > > Establishing a connection is not required if you are translating an IDoc. ## Settings Tab | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Connection** | Choose a connection from the list. | ### Select an Action When you add an SAP IDoc connector to a [flow](../../flows/flows), you must choose the action that the connector should perform. The **Settings** tab then updates to display the relevant options. The following table outlines the available actions. Click the links in the table to access the specific settings for each action. | Action | Description | | - | - | | [Send](./sap-idoc-send) | Connector settings specific to sending IDoc files to SAP. | | [Receive](./sap-idoc-receive) | Connector settings specific to receiving IDoc files from SAP. | | [Translate](./sap-idoc-translate) | Connector settings specific to translating IDoc files. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Action | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The amount of time before a failed send is retried. | | **Max Attempts** | The maximum number of times a failed send is retried. | ### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](./Settings-Page#alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Advanced Tab #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Other Settings #### Processing Delay
Setting Description
**Processing Delay** The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting.
*Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
## Logs Tab The **Logs** tab lists the service logs generated by this connector. Each row represents a log file and shows its date modified, file name, size, and status. Use the search bar or the status filter to find a specific log, click a file name to preview or download it, or click **Download All Logs** to download all of them as a single zip file. ## Establishing a Connection ### Configuring Services In addition to the [prerequisites](#prerequisites) described above, you must add the following services to `%WINDIR%\\system32\\drivers\\etc\\services`, or `/etc/services` if your system is running Linux or Mac OS X. If SAP NetWeaver is already installed on the machine, no further action is required. | Service Name | Connector Number/Protocol | | - | - | | sapdp00 | 3200/tcp | | sapdp01 | 3201/tcp | | sapdp02 | 3202/tcp | | sapdp03 | 3203/tcp | | sapdp04 | 3204/tcp | | sapdp05 | 3205/tcp | | sapdp06 | 3206/tcp | | sapdp07 | 3207/tcp | | sapdp08 | 3208/tcp | | sapdp09 | 3209/tcp | | sapdp10 | 3210/tcp | | sapdp11 | 3211/tcp | | sapdp12 | 3212/tcp | | sapdp13 | 3213/tcp | | sapdp14 | 3214/tcp | | sapdp15 | 3215/tcp | | sapdp16 | 3216/tcp | | sapdp17 | 3217/tcp | | sapdp18 | 3218/tcp | | sapdp19 | 3219/tcp | | sapdp20 | 3220/tcp | | sapdp21 | 3221/tcp | | sapdp22 | 3222/tcp | | sapdp23 | 3223/tcp | | sapdp24 | 3224/tcp | | sapdp25 | 3225/tcp | | sapdp26 | 3226/tcp | | sapdp27 | 3227/tcp | | sapdp28 | 3228/tcp | | sapdp29 | 3229/tcp | | sapdp30 | 3230/tcp | | sapdp31 | 3231/tcp | | sapdp32 | 3232/tcp | | sapdp33 | 3233/tcp | | sapdp34 | 3234/tcp | | sapdp35 | 3235/tcp | | sapdp36 | 3236/tcp | | sapdp37 | 3237/tcp | | sapdp38 | 3238/tcp | | sapdp39 | 3239/tcp | | sapdp40 | 3240/tcp | | sapdp41 | 3241/tcp | | sapdp42 | 3242/tcp | | sapdp43 | 3243/tcp | | sapdp44 | 3244/tcp | | sapdp45 | 3245/tcp | | sapdp46 | 3246/tcp | | sapdp47 | 3247/tcp | | sapdp48 | 3248/tcp | | sapdp49 | 3249/tcp | | sapdp50 | 3250/tcp | | sapdp51 | 3251/tcp | | sapdp52 | 3252/tcp | | sapdp53 | 3253/tcp | | sapdp54 | 3254/tcp | | sapdp55 | 3255/tcp | | sapdp56 | 3256/tcp | | sapdp57 | 3257/tcp | | sapdp58 | 3258/tcp | | sapdp59 | 3259/tcp | | sapdp60 | 3260/tcp | | sapdp61 | 3261/tcp | | sapdp62 | 3262/tcp | | sapdp63 | 3263/tcp | | sapdp64 | 3264/tcp | | sapdp65 | 3265/tcp | | sapdp66 | 3266/tcp | | sapdp67 | 3267/tcp | | sapdp68 | 3268/tcp | | sapdp69 | 3269/tcp | | sapdp70 | 3270/tcp | | sapdp71 | 3271/tcp | | sapdp72 | 3272/tcp | | sapdp73 | 3273/tcp | | sapdp74 | 3274/tcp | | sapdp75 | 3275/tcp | | sapdp76 | 3276/tcp | | sapdp77 | 3277/tcp | | sapdp78 | 3278/tcp | | sapdp79 | 3279/tcp | | sapdp80 | 3280/tcp | | sapdp81 | 3281/tcp | | sapdp82 | 3282/tcp | | sapdp83 | 3283/tcp | | sapdp84 | 3284/tcp | | sapdp85 | 3285/tcp | | sapdp86 | 3286/tcp | | sapdp87 | 3287/tcp | | sapdp88 | 3288/tcp | | sapdp89 | 3289/tcp | | sapdp90 | 3290/tcp | | sapdp91 | 3291/tcp | | sapdp92 | 3292/tcp | | sapdp93 | 3293/tcp | | sapdp94 | 3294/tcp | | sapdp95 | 3295/tcp | | sapdp96 | 3296/tcp | | sapdp97 | 3297/tcp | | sapdp98 | 3298/tcp | | sapdp99 | 3299/tcp | | sapgw00 | 3300/tcp | | sapgw01 | 3301/tcp | | sapgw02 | 3302/tcp | | sapgw03 | 3303/tcp | | sapgw04 | 3304/tcp | | sapgw05 | 3305/tcp | | sapgw06 | 3306/tcp | | sapgw07 | 3307/tcp | | sapgw08 | 3308/tcp | | sapgw09 | 3309/tcp | | sapgw10 | 3310/tcp | | sapgw11 | 3311/tcp | | sapgw12 | 3312/tcp | | sapgw13 | 3313/tcp | | sapgw14 | 3314/tcp | | sapgw15 | 3315/tcp | | sapgw16 | 3316/tcp | | sapgw17 | 3317/tcp | | sapgw18 | 3318/tcp | | sapgw19 | 3319/tcp | | sapgw20 | 3320/tcp | | sapgw21 | 3321/tcp | | sapgw22 | 3322/tcp | | sapgw23 | 3323/tcp | | sapgw24 | 3324/tcp | | sapgw25 | 3325/tcp | | sapgw26 | 3326/tcp | | sapgw27 | 3327/tcp | | sapgw28 | 3328/tcp | | sapgw29 | 3329/tcp | | sapgw30 | 3330/tcp | | sapgw31 | 3331/tcp | | sapgw32 | 3332/tcp | | sapgw33 | 3333/tcp | | sapgw34 | 3334/tcp | | sapgw35 | 3335/tcp | | sapgw36 | 3336/tcp | | sapgw37 | 3337/tcp | | sapgw38 | 3338/tcp | | sapgw39 | 3339/tcp | | sapgw40 | 3340/tcp | | sapgw41 | 3341/tcp | | sapgw42 | 3342/tcp | | sapgw43 | 3343/tcp | | sapgw44 | 3344/tcp | | sapgw45 | 3345/tcp | | sapgw46 | 3346/tcp | | sapgw47 | 3347/tcp | | sapgw48 | 3348/tcp | | sapgw49 | 3349/tcp | | sapgw50 | 3350/tcp | | sapgw51 | 3351/tcp | | sapgw52 | 3352/tcp | | sapgw53 | 3353/tcp | | sapgw54 | 3354/tcp | | sapgw55 | 3355/tcp | | sapgw56 | 3356/tcp | | sapgw57 | 3357/tcp | | sapgw58 | 3358/tcp | | sapgw59 | 3359/tcp | | sapgw60 | 3360/tcp | | sapgw61 | 3361/tcp | | sapgw62 | 3362/tcp | | sapgw63 | 3363/tcp | | sapgw64 | 3364/tcp | | sapgw65 | 3365/tcp | | sapgw66 | 3366/tcp | | sapgw67 | 3367/tcp | | sapgw68 | 3368/tcp | | sapgw69 | 3369/tcp | | sapgw70 | 3370/tcp | | sapgw71 | 3371/tcp | | sapgw72 | 3372/tcp | | sapgw73 | 3373/tcp | | sapgw74 | 3374/tcp | | sapgw75 | 3375/tcp | | sapgw76 | 3376/tcp | | sapgw77 | 3377/tcp | | sapgw78 | 3378/tcp | | sapgw79 | 3379/tcp | | sapgw80 | 3380/tcp | | sapgw81 | 3381/tcp | | sapgw82 | 3382/tcp | | sapgw83 | 3383/tcp | | sapgw84 | 3384/tcp | | sapgw85 | 3385/tcp | | sapgw86 | 3386/tcp | | sapgw87 | 3387/tcp | | sapgw88 | 3388/tcp | | sapgw89 | 3389/tcp | | sapgw90 | 3390/tcp | | sapgw91 | 3391/tcp | | sapgw92 | 3392/tcp | | sapgw93 | 3393/tcp | | sapgw94 | 3394/tcp | | sapgw95 | 3395/tcp | | sapgw96 | 3396/tcp | | sapgw97 | 3397/tcp | | sapgw98 | 3398/tcp | | sapgw99 | 3399/tcp | ### Logging on to SAP The following [settings](#settings-tab) are required to connect with your SAP system. | Setting | Description | | - | - | | **SAP Host** | Host name of the target system. Host names can be regular host names defined in a `hosts` file, an IP address like `123.123.123.123`, or an SAProuter address such as `/H/hostname/S/connector/H/host/S/connector/...` | | **System Number** | The number by which the target system is defined. The valid range is 0 to 99, but in general this is 0. | | **Client** | The client authenticating to the SAP system. | | **User** | The user that is authenticating to the SAP system. | | **Password** | The password for the authenticating user. | #### Gateway When connecting to SAP via a gateway, use the options in the [Gateway](#gateway) section of the **Advanced** tab. #### Load Balancing To log on via load balancing, set the properties in the [Logon Load Balancing](#logon-load-balancing) section of the **Advanced** tab. #### Secure Network Communications (SNC) To enable secure network communications, check the **Enable SNC** box in the [Secure Network Communications (SNC)](#secure-network-communications-snc) section of the **Advanced** tab. Set **SNC Library** to the .dll of the SNC library you want to use to connect with SAP. ### Print View When you view an SAP IDoc document on the connector **Transactions** tab, you can view it in a print preview format, or as the raw data. By default, the document is shown in the preview format, as shown in the following image: ![SAP IDoc print preview](/26.4/images/idoc_print_preview.png) The preview format shows the data in an easy-to-read layout. Use the **Raw View** toggle in the top right corner to view the unformatted document. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# SAP IDoc Receive Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sap-idoc/sap-idoc-receive Configuration for receiving IDoc files from SAP using the Arc SAP IDoc connector. ## Connector Configuration The following settings are available when you are receiving IDocs from SAP. #### Server Status *The state of the server.* | Status | Description | | - | - | | **Active** | The server is running. Click **Stop Server** to stop it. | | **Inactive** | The server is not running. Click **Start Server** to start it. | To review the logs the server produces while it runs, see the [Logs tab](./sap-idoc#logs-tab). #### Receiver Options *Settings related to receiving IDoc files.* | Setting | Description | | - | - | | **IDoc Receiver** | Whether the connector should retrieve IDoc files from SAP. | | **Program ID** | A unique name that identifies CData Arc to the SAP system. | | **IDoc Format** | Whether to receive files as raw IDoc or IDoc XML. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Refresh Schema** | When enabled, the connector attempts to update the schema it uses for parsing when a parsing error occurs. The schema is updated by reaching out to the SAP system and querying for the appropriate schema. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | #### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.3/self-hosted/en/getting-started/administration/settings/encryption-at-rest). | | **Sent Folder Scheme** (Deprecated) | Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | #### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab. | #### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Receiving IDocs from SAP Once you have configured the IDoc connector to connect to your SAP system, you can begin receiving IDoc files from SAP. The IDoc connector can receive files formatted in raw IDoc or XML IDoc. To receive files from SAP, create an RFC destination for Arc. You can then check **Enable IDoc Receiver** in the [Receiver Options](#receiver-options) section of the Settings tab, then enter the **Program Id** that you registered in your SAP system for Arc. ### Test Receiving an IDoc from SAP This example describes how to configure your SAP system to send a MATMAS IDoc to Arc. 1. *Define a logical system:* First, go to transaction SALE. Then navigate to Basic Settings > Logical Systems > Define Logical System. Create a new entry with a valid name (for example, Arc) and description. 2. *Create an RFC destination:* Open transaction code SM59. Click the create button and fill in the fields for the RFC destination described below. | Setting | Description | | - | - | | **RFC Destination** | A unique name for the new destination (for example, Arc\_DEST). | | **Connection Type** | **T** for TCP/IP. | | **Description** | A description for the destination. | | **Program Id** | A unique name that will be set in Arc to identify the destination. | | **Activation Type** | Select **Registered Server Program**. | ![SAP RFC destination settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-dest.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=bd6359dc29a92dd33d30f5d1b71e2404) After you save the settings, test the connection to make sure you can successfully connect to Arc. 3. *Create a processing connector:* Go to transaction code WE21. Create a new Transactional RFC connector with a unique name and the same destination you created in step 2. ![SAP receive connector settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-connector.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=fb8abea96b6f80741b65c07d1cb667b3) 4. *Create a partner profile:* Open transaction WE20. Create a partner profile for the logical system that you added in step 1. Next, add an outbound parameter for the MATMAS document. Set the **Message Type** to **MATMAS**. On the **Outbound Options** tab, set the **Receiver port** to the value of the connector you configured in step 3, set the **Output Mode** to **Transfer IDoc Immed**, and set the IDoc **Basic type** to **MATMAS05**. ![SAP receive partner profile settings](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-param.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=6a6bb3a880781c864d0a0ec799cd350d) 5. *Send a test IDoc:* Use transaction BD10 to send a test IDoc to Arc for a specified material number. You need to configure the **Message Type** and **Logical System** before sending the message. Use transaction BD87 to monitor outgoing messages and process them again if necessary. # SAP IDoc Send Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sap-idoc/sap-idoc-send Configuration for sending IDoc files to SAP using the Arc SAP IDoc connector. ## Connector Configuration The following settings are available when you are sending IDoc files to SAP. #### Control Record Information Options When you are receiving IDoc files from one SAP system and sending them to another SAP system you might want to change the control record information without changing the IDoc file data. See [Sending IDocs to SAP](#sending-idocs-to-sap) for more information. To overwrite the control headers for IDoc files, check **Overwrite**, then make your changes to the following options. | Setting | Description | | - | - | | **SAP Release (DOCREL)** | The SAP Release for IDOC. | | **Test Flag (TEST)** | The flag which indicates whether the IDoc should be in test mode. | | **Sender Port (SNDPOR)** | The sender port (SAP System or external subsystem). | | **Sender Partner Number (SNDPRN)** | The sender's partner number. | | **Sender Partner Type (SNDPRT)** | The sender's partner type. | | **Sender Partner Function (SNDPFC)** | The sender's partner function. | | **Sender Address (SNDSAD)** | The sender's address. | | **Logical Address (SNDLAD)** | The sender's logical address. | | **Receiver Port (RCVPOR)** | The receiver port. | | **Receiver Partner Number (RCVPRN)** | The receiver's partner number. | | **Receiver Partner Type (RCFPRT)** | The receiver's partner type. | | **Receiver Partner Function (RCVPFC)** | The receiver's partner function. | | **Recipient Address (RVCSAD)** | The receiver's address. | | **Receiver Logical Address (RCVLAD)** | The receiver's logical address. | | **Use System Date/Time** | Whether to include the system date/time in the control record information. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Refresh Schema** | When enabled, the connector attempts to update the schema it uses for parsing when a parsing error occurs. The schema is updated by reaching out to the SAP system and querying for the appropriate schema. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | #### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.3/self-hosted/en/getting-started/administration/settings/encryption-at-rest). | | **Sent Folder Scheme** (Deprecated) | Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | #### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab. | #### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Sending IDocs to SAP Once you have configured the IDoc connector to connect to your SAP system, you can begin sending IDoc files to SAP. The IDoc connector can send files formatted in raw IDoc or XML IDoc. You can configure the connector to automatically insert control record information into the IDoc being sent to SAP: the DOCREL, SNDPOR, SNDPRT, and so on. This makes it easier to process outgoing IDocs to the configured system. Configure these settings in the [Control Record Information Options](#control-record-information-options) section of the **Settings** tab. Before sending an IDoc to your SAP system, make sure that you have properly configured a partner profile in SAP. If no partner profile in SAP matches the control record information in the IDoc you are sending, the data will not be processed by your SAP system. See [Test Sending an IDoc to SAP](#test-sending-an-idoc-to-sap) for more information on setting up a partner profile. > [!TIP] > > You can use transaction code WE60 to search for segment documentation for different IDoc types. ### Test Sending an IDoc to SAP To test sending an IDoc to SAP, you first need to configure your SAP system to have a partner profile for Arc. The following steps describe how to do this with an ORDERS IDoc. 1. *Create a partner:* Go to transaction code WE20 and define a partner profile. Set **Partn Type** to **KU** for Customer and **Partner No** to a valid customer number. ![Create a Partner](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-send-profiles.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=3ec70123082bf6ae98e77e418eeeba0e) 2. *Add an inbound parameter for the ORDERS document:* The message type for this document is ORDERS. On the **Inbound Options** tab, set **Process code** to **ORDE** to tell SAP how to create a sales order when a purchase order ORDERS document is received by the SAP system. ![Set Parameters](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-send-params.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=a903f43e35a1a6aa840dbb7faee49c57) 3. *Assign the customer to a sales division:* The customer configured in step one must be assigned to a sales organization, distribution channel, and division. Go to transaction VOE2 to configure these settings in the EDSDC table. # SAP IDoc Translate Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sap-idoc/sap-idoc-translate Configuration for translating IDocs to XML and XML to IDocs using the Arc SAP IDoc connector. ## Connector Configuration The following settings are available when you are translating IDocs into XML, and vice versa. #### Control Record Information Options When you are translating IDoc files into XML, or translating XML files into IDoc, you might want to change the control record information without changing the IDoc file data. See [Sending IDocs to SAP](./sap-idoc-send#sending-idocs-to-sap) for more information. To overwrite the control headers for IDoc files, check **Overwrite**, then make your changes to the following options. | Setting | Description | | - | - | | **SAP Release (DOCREL)** | The SAP Release for IDOC. | | **Test Flag (TEST)** | The flag which indicates whether the IDoc should be in test mode. | | **Sender Port (SNDPOR)** | The sender port (SAP System or external subsystem). | | **Sender Partner Number (SNDPRN)** | The sender's partner number. | | **Sender Partner Type (SNDPRT)** | The sender's partner type. | | **Sender Partner Function (SNDPFC)** | The sender's partner function. | | **Sender Address (SNDSAD)** | The sender's address. | | **Logical Address (SNDLAD)** | The sender's logical address. | | **Receiver Port (RCVPOR)** | The receiver port. | | **Receiver Partner Number (RCVPRN)** | The receiver's partner number. | | **Receiver Partner Type (RCFPRT)** | The receiver's partner type. | | **Receiver Partner Function (RCVPFC)** | The receiver's partner function. | | **Recipient Address (RVCSAD)** | The receiver's address. | | **Receiver Logical Address (RCVLAD)** | The receiver's logical address. | | **Use System Date/Time** | Whether to include the system date/time in the control record information. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Refresh Schema** | When enabled, the connector attempts to update the schema it uses for parsing when a parsing error occurs. The schema is updated by reaching out to the SAP system and querying for the appropriate schema. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Upload Schema** | Use this to upload a schema and install it in the connector's Schema folder. If a schema already exists, you are asked if you want to overwrite it. | #### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.3/self-hosted/en/getting-started/administration/settings/encryption-at-rest). | | **Sent Folder Scheme** (Deprecated) | Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | #### Logging *Settings that govern the creation and storage of logs.* | Setting | Description | | - | - | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance. | | **Log Messages** | Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab. | #### Miscellaneous *Miscellaneous settings are for specific use cases.* | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Translate without a Connection to SAP You can translate IDocs to XML, and vice versa, without having a direct connection to SAP by uploading an additional document schema to the connector using the **Upload Schema** option in [Advanced Settings](#advanced-settings). The schema includes all elements of the base schema. To download the schema from SAP: * In SAP, find the documentation transaction with TCODE `WE60`. * If your IDoc type is basic, choose the **Basic type** radio button. Otherwise, choose **Enhancement**. * Click the **Documentation** menu and choose **XML Schema**. * From the **XML** menu, choose **Download**. * Save the downloaded file as `${IDocType}.xsd` (for example, `SHPMNT05.xsd`). * Upload that file to the connector using **Upload Schema**. # SAP IDoc Connector V1 (Deprecated) Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/sapidoc-v1 Send and receive SAP IDoc files using the SAP IDoc V1 connector with NetWeaver RFC, Classic RFC, JCO, or SOAP connection methods. The SAP IDoc connector supports sending and receiving IDoc files to and from a variety of SAP systems. It is a data container that is used to exchange information between SAP and another process that can understand the syntax and semantics of the data. ## Overview IDoc connectors can connect to SAP using the following methods: **Windows** * NetWeaver RFC SDK * Classic RFC SDK * Classic RFC SDK with Unicode support * SOAP (send-only) **Java** * Java connector (JCO) * SOAP (send-only) IDoc connectors can send and retrieve files formatted either as raw IDoc or as IDoc XML. ## Connector Configuration This section contains all of the configurable connector properties. ### Settings Tab #### Host Configuration *Settings related to establishing the connection to the SAP system.* | Setting | Description | | - | - | | **Connector Id** | The static, unique identifier for the connector. | | **Connector Type** | Displays the connector name and a description of what it does. | | **Connector Description** | An optional field to provide a free-form description of the connector and its role in the flow. | | **Connect Type** | The method with which to connect to SAP. See the [Establishing a Connection](#establishing-a-connection) section for your method for additional information. | | **SAP Host** | Hostname or IP address of the target SAP system. | | **System Number** | The number by which the target SAP system is defined. Valid values are 0-99, and often this value should be 0. | #### Client Authentication *Settings related to authenticating against the SAP system.* | Setting | Description | | - | - | | **Client** | The client that is authenticating to the SAP system. | | **User** | The username credential for authentication. | | **Password** | The password for the specified user. | #### Receiver Options *Settings related to receiving IDoc files.* | Setting | Description | | - | - | | **IDoc Receiver** | Whether the connector should retrieve IDoc files from SAP. | | **Program ID** | A unique name that identifies Arc to the SAP system. | | **IDoc Format** | Whether to receive files as raw IDoc or IDoc XML. | #### Control Record Information Options This section allows you to overwrite the control headers for IDoc files before sending them to SAP. This is useful when you are receiving IDoc files from one SAP system and sending them to another SAP system: in this situation you should change the control record without changing the IDoc file data. See [Sending IDocs to SAP](#sending-idocs-to-sap) for more information. ### Advanced Tab #### Language Settings *Settings related to communication language.* | Setting | Description | | - | - | | **Language** | The language value to be used when connecting to the SAP system. Set this to the ISO 639-1 value for the language the SAP system uses. | #### Gateway *Settings related to communicating with SAP through a gateway.* | Setting | Description | | - | - | | **Gateway Host** | The hostname or IP address of the gateway through which to connect. | | **Gateway Service** | The gateway service to use. | #### Logon Load Balancing *Settings related to logging in to the SAP system as a group to enable load balancing.* | Setting | Description | | - | - | | **Message Server** | The service name of the load balancer. | | **System ID** | Identifies the SAP system. | | **Logon Group** | The group to use when load balancing. | #### Secure Network Communications (SNC) *Settings related to using an external security product to enhance the SAP connection security.* | Setting | Description | | - | - | | **SNC Name** | The SNC name for the machine making the connection. | | **SNC Partner Name** | The SNC name of the SAP system the connector is connecting to. | | **SNC Quality of Protection Level** | The level of protection added by the SNC product. | | **SNC Library** | The path to the external security product's library. | #### Advanced Settings *Settings not included in the previous categories.* | Setting | Description | | - | - | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the **Transactions** tab is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Refresh Schema** | When enabled, the connector attempts to update the schema it uses for parsing when a parsing error occurs. The schema is updated by reaching out to the SAP system and querying for the appropriate schema. | #### Message *Message settings determine how the connector searches for messages and manages them after processing.* **Note:** The settings below are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](/26.4/self-hosted/en/connectors/file).
SettingDescription
**Save to Sent Folder** (Deprecated) Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by [Encryption at Rest](/26.4/self-hosted/en/getting-started/administration/settings/encryption-at-rest).
**Sent Folder Scheme** (Deprecated) Groups files in the **Sent** folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled.
#### Logging *Settings that govern the creation and storage of logs.*
Setting Description
**Log Level** The verbosity of logs generated by the connector. When you request support, set this to **Debug**.
**Log Subfolder Scheme** Instructs the connector to group files in the Logs folder according to the selected interval. The **Weekly** option (which is the default) instructs the connector to create a new subfolder each week and store all logs for the week in that folder. Leaving this setting blank tells the connector to save all logs directly in the Logs folder. For connectors that process many transactions, using subfolders helps keep logs organized and improves performance.
**Log Messages** Check this to have the log entry for a processed file include a copy of the file itself. If you disable this, you might not be able to download a copy of the file from the **Transactions** tab.
#### Miscellaneous *Miscellaneous settings are for specific use cases.*
Setting Description
**Other Settings** Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings.
### Automation Tab #### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The amount of time before a failed send is retried. | | **Max Attempts** | The maximum number of times a failed send is retried. | #### Performance *Settings related to the allocation of resources to the connector.*
Setting Description
**Max Workers** The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
**Max Files** The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page.
### Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory). Check **Allow ArcScript in Subject** to use ArcScript in the **Subject** field. When you select this, the **ArcScript Editor** button appears (![arcscript editor button](/26.4/images/rest_arcscript_editor.png)). Optionally, enter a comma-separated list of **Recipient** emails. ### SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. CData Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. SLA transitions to *At Risk* or *Violated* are also written to the [application log](/26.4/self-hosted/en/getting-started/administration/activity#application-logs), which means that Application Performance Monitoring (APM) and log-ingestion tools can also alert on them. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![SLA Empty](/26.4/images/sla_empty.png)
  • If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to.
  • In the **Expect at least** portion of the window:
    • Set the minimum number of transactions you expect to be processed (the volume)
    • Use the **Every** fields to specify the time frame
    • Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields.
    • Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary.
  • In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk.
    • By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**.
The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](/26.4/images/sla_defined.png) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Establishing a Connection ### Using NetWeaver RFC SDK The following libraries from the NetWeaver RFC SDK must be placed into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems): * sapnwrfc.dll * icudt30.dll * icuin30.dll * icuuc30.dll * libicudecnumber.dll * libsapucum.dll > [!NOTE] > > * The .dll names might be different based on the version of the SAP RFC SDK you have installed. The examples above are for version 3.0. > > * You need to add the `sapgw00` service to your listed services for the NetWeaver SDK connection to work on your client machine. You can add it to the services file at `C:\\Windows\\System32\\drivers\\etc\\services`. ### Using Classic RFC SDK The non-Unicode library does not require anything other than librfc32.dll. Place librfc32.dll into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems). > [!NOTE] > > These libraries are provided in both x86 and x64, so you must use the same platform that the process is using. If the application is running on a 64-bit version of Windows, the process is 64-bit. ### Using Classic RFC SDK with Unicode Support The following libraries from the SAP RFC SDK must be placed into the Arc `www\\bin\\x64` folder (`www\\bin\\x86` on 32-bit operating systems): * librfc32u.dll * icudt30.dll * icuin30.dll * icuuc30.dll > [!NOTE] > > * The .dll names might be different based on the version of the SAP RFC SDK you have installed. The examples above are for version 3.0. > > * These libraries are provided in both x86 and x64, so you must use the same platform that the process is using. If the application is running on a 64-bit version of Windows, the process is 64-bit. ### Using the Java Connector To use the Java connector, add sapjco3.jar to the `lib` folder of your Java servlet container. ### Using SOAP SOAP can only be used to *send* IDocs to SAP. You must use a different connection type to *receive* IDocs from SAP. > [!NOTE] > > This requires enabling SOAP connections on the SAP system. ### Configuring Services In addition to the dependencies above, the following services must be added to `%WINDIR%\\system32\\drivers\\etc\\services`, or `/etc/services` if your system is running Linux or Mac OS X. If SAP NetWeaver is already installed on the machine, no further action is required. | Service Name | Connector Number/Protocol | | - | - | | sapdp00 | 3200/tcp | | sapdp01 | 3201/tcp | | sapdp02 | 3202/tcp | | sapdp03 | 3203/tcp | | sapdp04 | 3204/tcp | | sapdp05 | 3205/tcp | | sapdp06 | 3206/tcp | | sapdp07 | 3207/tcp | | sapdp08 | 3208/tcp | | sapdp09 | 3209/tcp | | sapdp10 | 3210/tcp | | sapdp11 | 3211/tcp | | sapdp12 | 3212/tcp | | sapdp13 | 3213/tcp | | sapdp14 | 3214/tcp | | sapdp15 | 3215/tcp | | sapdp16 | 3216/tcp | | sapdp17 | 3217/tcp | | sapdp18 | 3218/tcp | | sapdp19 | 3219/tcp | | sapdp20 | 3220/tcp | | sapdp21 | 3221/tcp | | sapdp22 | 3222/tcp | | sapdp23 | 3223/tcp | | sapdp24 | 3224/tcp | | sapdp25 | 3225/tcp | | sapdp26 | 3226/tcp | | sapdp27 | 3227/tcp | | sapdp28 | 3228/tcp | | sapdp29 | 3229/tcp | | sapdp30 | 3230/tcp | | sapdp31 | 3231/tcp | | sapdp32 | 3232/tcp | | sapdp33 | 3233/tcp | | sapdp34 | 3234/tcp | | sapdp35 | 3235/tcp | | sapdp36 | 3236/tcp | | sapdp37 | 3237/tcp | | sapdp38 | 3238/tcp | | sapdp39 | 3239/tcp | | sapdp40 | 3240/tcp | | sapdp41 | 3241/tcp | | sapdp42 | 3242/tcp | | sapdp43 | 3243/tcp | | sapdp44 | 3244/tcp | | sapdp45 | 3245/tcp | | sapdp46 | 3246/tcp | | sapdp47 | 3247/tcp | | sapdp48 | 3248/tcp | | sapdp49 | 3249/tcp | | sapdp50 | 3250/tcp | | sapdp51 | 3251/tcp | | sapdp52 | 3252/tcp | | sapdp53 | 3253/tcp | | sapdp54 | 3254/tcp | | sapdp55 | 3255/tcp | | sapdp56 | 3256/tcp | | sapdp57 | 3257/tcp | | sapdp58 | 3258/tcp | | sapdp59 | 3259/tcp | | sapdp60 | 3260/tcp | | sapdp61 | 3261/tcp | | sapdp62 | 3262/tcp | | sapdp63 | 3263/tcp | | sapdp64 | 3264/tcp | | sapdp65 | 3265/tcp | | sapdp66 | 3266/tcp | | sapdp67 | 3267/tcp | | sapdp68 | 3268/tcp | | sapdp69 | 3269/tcp | | sapdp70 | 3270/tcp | | sapdp71 | 3271/tcp | | sapdp72 | 3272/tcp | | sapdp73 | 3273/tcp | | sapdp74 | 3274/tcp | | sapdp75 | 3275/tcp | | sapdp76 | 3276/tcp | | sapdp77 | 3277/tcp | | sapdp78 | 3278/tcp | | sapdp79 | 3279/tcp | | sapdp80 | 3280/tcp | | sapdp81 | 3281/tcp | | sapdp82 | 3282/tcp | | sapdp83 | 3283/tcp | | sapdp84 | 3284/tcp | | sapdp85 | 3285/tcp | | sapdp86 | 3286/tcp | | sapdp87 | 3287/tcp | | sapdp88 | 3288/tcp | | sapdp89 | 3289/tcp | | sapdp90 | 3290/tcp | | sapdp91 | 3291/tcp | | sapdp92 | 3292/tcp | | sapdp93 | 3293/tcp | | sapdp94 | 3294/tcp | | sapdp95 | 3295/tcp | | sapdp96 | 3296/tcp | | sapdp97 | 3297/tcp | | sapdp98 | 3298/tcp | | sapdp99 | 3299/tcp | | sapgw00 | 3300/tcp | | sapgw01 | 3301/tcp | | sapgw02 | 3302/tcp | | sapgw03 | 3303/tcp | | sapgw04 | 3304/tcp | | sapgw05 | 3305/tcp | | sapgw06 | 3306/tcp | | sapgw07 | 3307/tcp | | sapgw08 | 3308/tcp | | sapgw09 | 3309/tcp | | sapgw10 | 3310/tcp | | sapgw11 | 3311/tcp | | sapgw12 | 3312/tcp | | sapgw13 | 3313/tcp | | sapgw14 | 3314/tcp | | sapgw15 | 3315/tcp | | sapgw16 | 3316/tcp | | sapgw17 | 3317/tcp | | sapgw18 | 3318/tcp | | sapgw19 | 3319/tcp | | sapgw20 | 3320/tcp | | sapgw21 | 3321/tcp | | sapgw22 | 3322/tcp | | sapgw23 | 3323/tcp | | sapgw24 | 3324/tcp | | sapgw25 | 3325/tcp | | sapgw26 | 3326/tcp | | sapgw27 | 3327/tcp | | sapgw28 | 3328/tcp | | sapgw29 | 3329/tcp | | sapgw30 | 3330/tcp | | sapgw31 | 3331/tcp | | sapgw32 | 3332/tcp | | sapgw33 | 3333/tcp | | sapgw34 | 3334/tcp | | sapgw35 | 3335/tcp | | sapgw36 | 3336/tcp | | sapgw37 | 3337/tcp | | sapgw38 | 3338/tcp | | sapgw39 | 3339/tcp | | sapgw40 | 3340/tcp | | sapgw41 | 3341/tcp | | sapgw42 | 3342/tcp | | sapgw43 | 3343/tcp | | sapgw44 | 3344/tcp | | sapgw45 | 3345/tcp | | sapgw46 | 3346/tcp | | sapgw47 | 3347/tcp | | sapgw48 | 3348/tcp | | sapgw49 | 3349/tcp | | sapgw50 | 3350/tcp | | sapgw51 | 3351/tcp | | sapgw52 | 3352/tcp | | sapgw53 | 3353/tcp | | sapgw54 | 3354/tcp | | sapgw55 | 3355/tcp | | sapgw56 | 3356/tcp | | sapgw57 | 3357/tcp | | sapgw58 | 3358/tcp | | sapgw59 | 3359/tcp | | sapgw60 | 3360/tcp | | sapgw61 | 3361/tcp | | sapgw62 | 3362/tcp | | sapgw63 | 3363/tcp | | sapgw64 | 3364/tcp | | sapgw65 | 3365/tcp | | sapgw66 | 3366/tcp | | sapgw67 | 3367/tcp | | sapgw68 | 3368/tcp | | sapgw69 | 3369/tcp | | sapgw70 | 3370/tcp | | sapgw71 | 3371/tcp | | sapgw72 | 3372/tcp | | sapgw73 | 3373/tcp | | sapgw74 | 3374/tcp | | sapgw75 | 3375/tcp | | sapgw76 | 3376/tcp | | sapgw77 | 3377/tcp | | sapgw78 | 3378/tcp | | sapgw79 | 3379/tcp | | sapgw80 | 3380/tcp | | sapgw81 | 3381/tcp | | sapgw82 | 3382/tcp | | sapgw83 | 3383/tcp | | sapgw84 | 3384/tcp | | sapgw85 | 3385/tcp | | sapgw86 | 3386/tcp | | sapgw87 | 3387/tcp | | sapgw88 | 3388/tcp | | sapgw89 | 3389/tcp | | sapgw90 | 3390/tcp | | sapgw91 | 3391/tcp | | sapgw92 | 3392/tcp | | sapgw93 | 3393/tcp | | sapgw94 | 3394/tcp | | sapgw95 | 3395/tcp | | sapgw96 | 3396/tcp | | sapgw97 | 3397/tcp | | sapgw98 | 3398/tcp | | sapgw99 | 3399/tcp | ### Logging on to SAP The following [settings](#settings-tab) are required to connect with your SAP system. | Setting | Description | | - | - | | **SAP Host** | Host name of the target system. Host names can be regular host names defined in a `hosts` file, an IP address like `123.123.123.123`, or an SAProuter address such as `/H/hostname/S/connector/H/host/S/connector/...` | | **System Number** | The number by which the target system is defined. The valid range is 0 to 99, but in general this is 0. | | **Client** | The client authenticating to the SAP system. | | **User** | The user that is authenticating to the SAP system. | | **Password** | The password for the authenticating user. | #### Gateway When connecting to SAP via a gateway, use the options in the [Gateway](#gateway) section of the **Advanced** tab. #### Load Balancing To log on via load balancing, set the properties in the [Logon Load Balancing](#logon-load-balancing) section of the **Advanced** tab. #### Secure Network Communications (SNC) To enable secure network communications, check the **Enable SNC** box in the [Secure Network Communications (SNC)](#secure-network-communications-snc) section of the **Advanced** tab. Set **SNC Library** to the .dll of the SNC library you want to use to connect with SAP. ## Sending IDocs to SAP Once you have configured the IDoc connector to connect to your SAP system, you can begin sending IDoc files to SAP. The IDoc connector can send files formatted in raw IDoc or XML IDoc. You can configure the connector to automatically insert control record information into the IDoc being sent to SAP: the DOCREL, SNDPOR, SNDPRT, and so on. This makes it easier to process outgoing IDocs to the configured system. Configure these settings in the [Control Record Information Options](#control-record-information-options) section of the **Settings** tab. Before sending an IDoc to your SAP system, make sure that you have properly configured a partner profile in SAP. If no partner profile in SAP matches the control record information in the IDoc you are sending, the data will not be processed by your SAP system. See [Test Sending an IDoc to SAP](#test-receiving-an-idoc-from-sap) for more information on setting up a partner profile. > [!TIP] > > You can use transaction code WE60 to search for segment documentation for different IDoc types. ### Test Sending an IDoc to SAP To test sending an IDoc to SAP, you first need to configure your SAP system to have a partner profile for Arc. The following steps describe how to do this with an ORDERS IDoc. 1. *Create a partner:* Go to transaction code WE20 and define a partner profile. Set **Partn Type** to **KU** for Customer and **Partner No** to a valid customer number. ![Create a Partner](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-send-profiles.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=3ec70123082bf6ae98e77e418eeeba0e) 2. *Add an inbound parameter for the ORDERS document:* The message type for this document is ORDERS. On the **Inbound Options** tab, set **Process code** to **ORDE** to tell SAP how to create a sales order when a purchase order ORDERS document is received by the SAP system. ![Set Parameters](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-send-params.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=a903f43e35a1a6aa840dbb7faee49c57) 3. *Assign the customer to a sales division:* The customer configured in step one must be assigned to a sales organization, distribution channel, and division. Go to transaction VOE2 to configure these settings in the EDSDC table. ## Receiving IDocs from SAP Once you have configured the IDoc connector to connect to your SAP system, you can begin receiving IDoc files from SAP. The IDoc connector can receive files formatted in raw IDoc or XML IDoc. To receive files from SAP, create an RFC destination for Arc. You can then check **Enable IDoc Receiver** in the [Receiver Options](#receiver-options) section of the Settings tab, then enter the **Program Id** that you registered in your SAP system for Arc. ### Test Receiving an IDoc from SAP This example describes how to configure your SAP system to send a MATMAS IDoc to Arc. 1. *Define a logical system:* First, go to transaction SALE. Then navigate to Basic Settings > Logical Systems > Define Logical System. Create a new entry with a valid name (for example, Arc) and description. 2. *Create an RFC destination:* Open transaction code SM59. Click the create button and fill in the fields for the RFC destination described below. | Setting | Description | | - | - | | **RFC Destination** | A unique name for the new destination (for example, Arc\_DEST). | | **Connection Type** | **T** for TCP/IP. | | **Description** | A description for the destination. | | **Program Id** | A unique name that will be set in Arc to identify the destination. | | **Activation Type** | Select **Registered Server Program**. | ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-dest.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=bd6359dc29a92dd33d30f5d1b71e2404) After you save the settings, test the connection to make sure you can successfully connect to Arc. 3. *Create a processing connector:* Go to transaction code WE21. Create a new Transactional RFC connector with a unique name and the same destination you created in step 2. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-connector.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=fb8abea96b6f80741b65c07d1cb667b3) 4. *Create a partner profile:* Open transaction WE20. Create a partner profile for the logical system that you added in step 1. Next, add an outbound parameter for the MATMAS document. Set the **Message Type** to **MATMAS**. On the **Outbound Options** tab, set the **Receiver port** to the value of the connector you configured in step 3, set the **Output Mode** to **Transfer IDoc Immed**, and set the IDoc **Basic type** to **MATMAS05**. ![](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sap-rec-param.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=6a6bb3a880781c864d0a0ec799cd350d) 5. *Send a test IDoc:* Use transaction BD10 to send a test IDoc to Arc for a specified material number. You need to configure the **Message Type** and **Logical System** before sending the message. Use transaction BD87 to monitor outgoing messages and process them again if necessary. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. CData Arc supports these macros, which all use the following syntax: `%Macro%`.
MacroDescription
ConnectorIDEvaluates to the ConnectorID of the connector.
ConnectorNameEvaluates to the name of the connector. Enables you to include the connection name in file names or paths: for example, to tag backup files by which database connection produced them.
ExtEvaluates to the file extension of the file currently being processed by the connector.
FilenameEvaluates to the filename (extension included) of the file currently being processed by the connector.
FilenameNoExtEvaluates to the filename (without the extension) of the file currently being processed by the connector.
MessageIdEvaluates to the MessageId of the message being output by the connector.
RegexFilename:*pattern*Applies a RegEx pattern to the filename of the file currently being processed by the connector.
Header:*headername*Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector.
LongDateEvaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024).
ShortDateEvaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24).
DateFormat:*format*Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats.
Vault:*vaultitem*Evaluates to the value of the specified vault item.
### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument:
  • %Header:headername%: Where `headername` is the name of a header on a message.
  • %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message.
  • %Header:ponum% resolves to the value of the `ponum` header set on the input message.
  • %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`).
  • %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault.
  • %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file.
You can also create more sophisticated macros, as shown in the following examples:
  • Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%`
  • Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%`
  • Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%`
# ShipStation Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation Connect Arc to ShipStation for data integration and synchronization The ShipStation connector allows you to integrate ShipStation into your data flow by pushing or pulling data from ShipStation. ## Key Capabilities * Shipping and fulfillment platform integration for e-commerce order management * API-based connectivity for shipment creation, tracking, label generation, and order management * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to ShipStation. ## Establish a Connection To allow Arc to use data from ShipStation, you must first establish a connection to ShipStation. There are two ways to establish this connection: * Add a ShipStation connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **ShipStation**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to ShipStation can be reused across multiple ShipStation connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to ShipStation. * **API Key** — The API key for your ShipStation account. * **API Secret** — The API secret for your ShipStation account. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to ShipStation with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to ShipStation, you must choose the action that the ShipStation connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Lookup](./shipstation-lookup) | Retrieves values from ShipStation and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from ShipStation. It should be formatted as a SQL query against the ShipStation tables. | Middle | | [Lookup Stored Procedure](./shipstation-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./shipstation-select) | Retrieves data from ShipStation and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./shipstation-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Create Shipment Label](./shipstation-create-shipment-label) | Creates a shipping label in ShipStation. | End | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./shipstation-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # ShipStation Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-ac Reference for advanced settings available on the ShipStation connector's Advanced tab. The **Advanced** tab allows you to configure advanced properties of the ShipStation connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The ShipStation connector models ShipStation tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and ShipStation values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The ShipStation connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into ShipStation. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The ShipStation connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the ShipStation connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by ShipStation. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by ShipStation, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from ShipStation. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure ShipStation connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # ShipStation Create Shipment Label Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-create-shipment-label Create a shipping label in ShipStation at the end of an Arc flow. The Create Shipment Label action creates a shipping label in ShipStation. This action belongs at the end of a flow. When creating a shipment label: * Set the package specifications, "ship to" values, and customs information [with an XML Map connector](#xml-map-connector-package-ship-to-customs). * Enter the "ship from" and test label values in the [connector configuration pane](#connector-configuration-ship-from). ## XML Map Connector: Package, Ship To, Customs You must connect an [XML Map](../xml-map/xml-map) connector to the ShipStation connector to map the package specifications, "ship to" values, and customs information. In the example below, the shipment information appears in the **Source** column, and the ShipStation shipping label parameters appear in the **Destination** column. ![ShipStation Label Mapping](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/shipstation_labelmapping.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=c178297929fc22a48f9004db98c853aa) ### Shipment Label Values The table below lists the values that you can map for shipment labels using the XML Map connector interface. | Name | Data Type | Required? | Description | | - | - | - | - | | CarrierCode | String | Yes | Shipping carrier code. | | ServiceCode | String | Yes | Shipping service code. | | PackageCode | String | Yes | Package type code. | | Confirmation | String | No | Type of delivery confirmation to use. Allowed values are **none**, **delivery**, **signature**, **adult\_signature**, or **direct\_signature**. **direct\_signature** is available for FedEx only. | | ShipDate | String | Yes | Date the order should be shipped. | | WeightValue | Integer | Yes | Weight value for the shipment. | | WeightUnits | String | Yes | Unit for the weight value. | | WeightWeightUnits | Integer | No | Weight Weight Units for the shipment. | | DimensionsLength | String | No | Package length. | | DimensionsWidth | String | No | Package width. | | DimensionsHeight | String | No | Package height. | | DimensionsUnits | String | No | Units of measurement for the dimensions. Allowed values are **inches** or **centimeters**. | | ShipToName | String | Yes | Name of person. | | ShipToCompany | String | Yes | Name of company. | | ShipToStreet1 | String | Yes | First line of address. | | ShipToStreet2 | String | No | Second line of address, if applicable. | | ShipToStreet3 | String | No | Third line of address, if applicable. | | ShipToCity | String | Yes | Name of city. | | ShipToState | String | Yes | Name of state. | | ShipToPostalCode | String | Yes | Postal code. | | ShipToCountry | String | Yes | Two-character ISO country code. | | ShipToPhone | String | No | Telephone number for recipient. | | ShipToResidential | Boolean | Yes | Whether the ShipTo address is residential. | | ShipToAddressVerified | String | No | Whether the address has been verified by ShipStation. Allowed values are **Address not yet validated**, **Address validated successfully**, **Address validation warning**, or **Address validation failed**. | | InsuranceOptions InsureShipment | String | No | Whether the shipment should be insured. | | InsuranceOptions Provider | String | No | Preferred insurance provider. Allowed values are **shipsurance**, **carrier**, or **provider**. **provider** indicates that a shipment was insured by a third party other than ShipSurance or the carrier. Provider insurance is handled outside of ShipStation and does not affect the cost of processing the label. | | InternationalOptions Contents | String | No | Contents for international shipments. Allowed values are **merchandise**, **documents**, **gift**, **returned\_goods**, or **sample**. | | InternationalOptions CustomsItems CustomsItemId | String | No | ItemId values for customs. If there are multiple values, separate them with commas (for example, **121221,21211,12121**). | | InternationalOptions CustomsItems Description | String | No | A short description of the items for customs. If you have multiple items, provide descriptions for each one by separating them with commas (for example, **desc1,desc2,desc3**). | | InternationalOptions CustomsItems Quantity | String | No | Quantity of each item for customs. If you have multiple items, provide quantities for each one by separating them with commas (for example, **5,1,4**). | | InternationalOptions CustomsItems Value | String | No | Value (in USD) of each item for customs. If you have multiple items, provide values for each one by separating them with commas (for example, **10,50,100**). | | InternationalOptions CustomsItems HarmonizedTariffCode | String | No | Harmonized Commodity Code of each item for customs. If you have multiple items, provide codes for each one by separating them with commas (for example, **10012,40021,30091**). | | InternationalOptions CustomsItems CountryOfOrigin | String | No | Two-character ISO country code of each item for customs. If you have multiple items, provide codes for each one by separating them with commas (for example, **US,US,US**). | | InternationalOptions NonDelivery | String | No | Non-delivery options for international shipments. Allowed values are **return\_to\_sender** or **treat\_as\_abandoned**. | | AdvancedOptions WarehouseId | Integer | No | The Id of the warehouse that the order will ship from. If the order is fulfilled using a fill provider, there is no warehouse attached and this value is **null**. | | AdvancedOptions NonMachinable | Boolean | No | Whether the order is non-machinable. | | AdvancedOptions SaturdayDelivery | Boolean | No | Whether the order is to be delivered on a Saturday. | | AdvancedOptions ContainsAlcohol | Boolean | No | Whether the order contains alcohol. | | AdvancedOptions StoreId | Integer | No | Id of the store associated with the order. If this value is not specified, ShipStation defaults to the first manual store on the account. | | AdvancedOptions CustomField1 | String | No | Field that allows for custom data to be associated with an order. | | AdvancedOptions CustomField2 | String | No | Field that allows for custom data to be associated with an order. | | AdvancedOptions CustomField3 | String | No | Field that allows for custom data to be associated with an order. | | AdvancedOptions Source | String | No | Original source or marketplace for the order. | | AdvancedOptions MergedOrSplit | Boolean | No | Whether an order has been merged with or split from another order. | | AdvancedOptions MergedIds | Integer | No | If the order was merged, this is an array of the orderIds that were merged. | | AdvancedOptions ParentId | Integer | No | If the order was split from a parent order, this is the orderId of the parent order. | | AdvancedOptions BillToParty | String | No | Party to bill for the shipment. Allowed values are **my\_account**, **my\_other\_account**, **recipient**, or **third\_party**. | | AdvancedOptions BillToAccount | String | No | Account number of the party to bill for the shipment. | | AdvancedOptions BillToPostalCode | String | No | Postal code of the party to bill for the shipment. | | AdvancedOptions BillToCountryCode | String | No | Two-character ISO country code of the party to bill for the shipment. | | AdvancedOptions BillToMyOtherAccount | String | No | If **AdvancedOptions BillToParty** is set to **my\_other\_account**, this is the shippingProviderId value for the desired account. You can make a *List Carriers* call to obtain shippingProviderId values. | ## Connector Configuration: Ship From The configuration pane for the Create Shipment Label connector allows you to specify the **ship from** information and toggle between test labels and real labels. The table below outlines these values: | Setting | Description | | - | - | | **Ship From Name** | Name of person. | | **Ship From Company** | Name of company. | | **Ship From Street 1** | First line of address. | | **Ship From Street 2** | Second line of address, if applicable. | | **Ship From Street 3** | Third line of address, if applicable. | | **Ship From City** | Name of city. | | **Ship From State** | Name of state. | | **Ship From Postal Code** | Postal code. | | **Ship From Country** | Two-character ISO country code. | | **Ship From Phone** | Telephone number for shipper. | | **Ship From Residential** | Whether the Ship From address is residential. | | **Ship From Address Verified** | Whether the address has been verified by ShipStation. Allowed values are **Address not yet validated**, **Address validated successfully**, **Address validation warning**, or **Address validation failed**. | | **Enable Test Mode** | Whether the shipping label should be a test label. Disabled by default. | # ShipStation Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-esp Treat data coming into the connector as input for a stored procedure, then pass the result down the flow. The Execute Stored Procedures action treats data coming into the ShipStation connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the ShipStation connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # ShipStation Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-lookup Retrieve values from ShipStation and insert them into an Arc message in the flow. The Lookup action retrieves values from ShipStation and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the ShipStation connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the ShipStation connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the ShipStation connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the ShipStation connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from ShipStation tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The ShipStation connector can perform Lookup queries against a local cache instead of querying ShipStation directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query ShipStation on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries ShipStation directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to ShipStation.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # ShipStation Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-lsp Treat data coming into the connector as input for a stored procedure, then insert the result into an existing Arc message in the flow. The Lookup Stored Procedure action treats data coming into the ShipStation connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the ShipStation connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the ShipStation connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# ShipStation Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shipstation/shipstation-select Retrieve data from ShipStation and bring it into Arc. The Select action retrieves data from ShipStation and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries ShipStation. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to ShipStation, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from ShipStation. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/shipstation/shipstation-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in ShipStation relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what ShipStation supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured ShipStation connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a ShipStation Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **ShipStation\_Select** connector pulls data from a ShipStation connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **ShipStation\_Select** then passes down the flow from **CSV1**.
# Shopify Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify Connect Arc to Shopify for data integration and synchronization The Shopify connector allows you to integrate Shopify into your data flow by pushing or pulling data from Shopify. ## Key Capabilities * E-commerce platform integration with OAuth and Access Token authentication * Shop URL-based connectivity for product, order, customer, and inventory data access * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Shopify. ## Establish a Connection To allow Arc to use data from Shopify, you must first establish a connection to Shopify. There are two ways to establish this connection: * Add a Shopify connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Shopify**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Shopify can be reused across multiple Shopify connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Shopify. * **Auth Scheme** — Choose your auth scheme from the list. Options are **OAuth** or **Access Token**. * **Shop URL** — The full URL of your Shopify shop. * **OAuth Client Id** — The client Id for your account (OAuth authentication only). * **OAuth Client Secret** — The client secret for your account (OAuth authentication only). * **Access Token** — The access token for establishing the connection (Access Token authentication only). 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Shopify with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Shopify, you must choose the action that the Shopify connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./shopify-upsert) | Inserts or updates Shopify data. By default, if a record already exists in Shopify, an update is performed on the existing data in Shopify using the values provided from the input. | End | | [Lookup](./shopify-lookup) | Retrieves values from Shopify and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Shopify. It should be formatted as a SQL query against the Shopify tables. | Middle | | [Lookup Stored Procedure](./shopify-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./shopify-select) | Retrieves data from Shopify and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./shopify-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./shopify-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Shopify Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-ac Advanced configuration options for the Shopify connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Shopify connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Shopify connector models Shopify tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Shopify values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Shopify connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Shopify. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Shopify connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Shopify connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Shopify. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Shopify, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Shopify. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Shopify connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Shopify Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Shopify connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Shopify connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Shopify Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-lookup Retrieve values from Shopify and insert them into an Arc message in the flow. The Lookup action retrieves values from Shopify and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Shopify connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Shopify connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Shopify connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Shopify connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Shopify tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Shopify connector can perform Lookup queries against a local cache instead of querying Shopify directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Shopify on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Shopify directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Shopify.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Shopify Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-lsp Use a stored procedure to retrieve values from Shopify and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Shopify connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Shopify connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Shopify connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Shopify Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-select Retrieve data from Shopify and bring it into Arc. The Select action retrieves data from Shopify and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Shopify. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector queries tables using a parent-child hierarchy. You can add additional tables as child tables or by using multiple connectors in your flow.
  • [Joined Tables](#joined-tables): The connector issues a single JOIN query that combines data from two or more related tables.
The structure types are described in the following sections. > [!NOTE] > > Joined Tables issues a single query to Shopify, which is more efficient than the Parent and Child Tables approach for data sources that enforce API rate limits. ### Parent and Child Tables Use the dropdown list to select a target table from Shopify. ![Dropdown for selecting a target table in the source structure editor](/26.4/images/dbsa_select_tableadd_263.png) Use the arrow to add child tables to your structure. The following image shows Bill selected as the parent table, and a child table ready to be selected. When you have finished with your selections, click **Apply**. ![Table list with Bill selected as the parent table and a child table ready to be selected](/26.4/images/dbsa_tablelist_263.png) #### Results After you click **Apply**, an **Items** node appears in the results structure. To configure the columns and filters for your query, click the ellipsis (**...**) on the table name node and choose **Table Settings**. The **Edit \[table name] Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit table Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from the selected tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > You can select more tables by clicking the link to your **Source** table name and returning to **Edit Source Structure** pane. See [Child Tables](/26.4/self-hosted/en/connectors/shopify/shopify-ac#child-tables) for more information. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the table designer and apply to the entire query.
  • **Only process new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ### Joined Tables When you select **Joined Tables** as the structure type, the JOIN designer appears. Use the JOIN designer to define how tables in Shopify relate to each other. The connector translates your configuration into a single JOIN query. ![JOIN designer with empty join row fields](/26.4/images/dbsa_join_designer.png) Each row in the JOIN designer represents one join relationship and contains the following fields:
  • **Join Type**: The type of SQL join to perform. The available options depend on what Shopify supports. Common options include **Inner Join**, **Left Join**, **Right Join**, and **Full Outer Join**.
  • **Left Table**: The table on the left side of the join. The first join row's left table is the base table for the query.
  • **Left Column**: The column from the left table to join on.
  • **Right Table**: The related table on the right side of the join.
  • **Right Column**: The column from the right table that matches the left column.
The following example joins a **Bill** table to a **Bill1** table on their shared **Id** column: ![JOIN designer with Bill and Bill1 tables joined on the Id column](/26.4/images/dbsa_join_addtable.png) This produces a query equivalent to: ```sql theme={null} SELECT ... FROM Bill INNER JOIN Bill1 ON Bill.Id = Bill1.Id ``` To join more than two tables, click **Add Join** to add another join row. Each additional row joins a new table to the result of the previous join. When you are done configuring the join, click **Apply**. #### Results After you click **Apply**, a **Results** node appears in the results structure. To configure the columns and filters for your join output, click the ellipsis (**...**) on the Results node and choose **Results Settings**. The **Edit Results Settings** dialog opens with two tabs: **Columns** and **Filters**. ![Edit Results Settings dialog showing the Columns tab with source table tags](/26.4/images/dbsa_join_results.png) In the **Columns** tab, all columns from all joined tables are listed. Each column displays a tag indicating its source table. By default, all columns are selected. Uncheck a column to exclude it from the query output. > [!NOTE] > > When two joined tables share a column name, the connector automatically prefixes the duplicate column with its table name to avoid collisions. At the bottom of the **Columns** tab, use the **Order by** field to sort the output by a column, and choose **ASC** (ascending) or **DESC** (descending) order. The **Filters** tab lets you add filters to limit the rows the connector selects. See [Filters](#filters) for more information. #### Options The **Options** panel on the right side of the **Edit Source Structure** dialog contains settings that control how the connector processes records. These settings are available alongside the JOIN designer and apply to the entire join query.
  • **Use column \[x] for processing new or changed records**: Enables incremental processing so that the connector only selects rows that are new or changed since the last run. In a Joined Tables configuration, this column must come from the base (leftmost) table. Columns from joined tables are not available for this setting.
  • **Update \[column] with \[value] when rows are processed**: Updates a column with a specified value after each row is processed.
You can also preview the output structure or view the underlying XML configuration by toggling **Preview XML** or **Code View** at the top of the **Edit Source Structure** dialog. ## Filters You can optionally add filters to limit the data the connector selects. Click the **Filters** tab in the settings dialog to configure filters. The connector translates your filters into query syntax. > [!NOTE] > > You do not need to apply a filter to your data if it does not suit your use case. ### Creating a Filter To create a filter, follow these steps:
  1. Click **Add Filter**.
  2. In the column dropdown, choose the column that you want to filter.
  3. In the condition dropdown, choose the filtering condition. The available conditions depend on the data type of the column you selected. For example, a numeric column has the condition **Less Than or Equal To**, whereas a string column has **Contains**.
  4. Enter a value in the value field. If the condition does not require a value (for example, **Is Null**), the value field is not shown.
The image below shows an example of a filter on a column named **Vendor**. With this filter in place, the connector only selects rows where the **Vendor** column contains **CData**. ![Filter configured on the Vendor column with Contains CData condition](/26.4/images/dbsa_filter.png) ### Filter Groups You can apply multiple filters by clicking **Add Filter** and creating more filters. All filters belong to the same group by default. Click **Add Group** to create a nested group and separate filters by different logic. Each group has the following options at the top:
  • **NOT** toggle: Inverts the logic of the entire group. For example, toggling NOT on a group that contains **Amount Less Than or Equal To 5000** instructs the connector to select only rows where the amount is greater than 5000.
  • **And / Or** toggle: Determines how this group relates to adjacent filters or groups.
    • If you select **And**, this group's condition must be true in addition to adjacent conditions.
    • If you select **Or**, this group's condition is an alternative to adjacent conditions.
The screenshot below shows an example with multiple filter groups. ![Multiple filter groups showing AND and OR logic](/26.4/images/dbsa_filter_group.png) In this example, the filter condition is met when all of the following are true:
  • The **Vendor** column contains **CData**
  • The **Status** column equals **Active** or **Pending**
  • The **Amount** column is *not* less than or equal to **5000** (that is, Amount is greater than 5000)
If this filter condition is met, the connector selects the matching data and passes it down the flow. ## Example Arc Flow After you choose the tables and columns for the Select-configured Shopify connector, place the connector at the beginning of an Arc flow. The screenshot below depicts an example flow with a Shopify Select connector, an XML Map connector, a CSV connector, and a File connector at the end: ![](/26.4/images/dbsa_select_flow.png) Data in this example flow passes through these steps:
  1. The **Shopify\_Select** connector pulls data from a Shopify connection.
  2. This data passes to the **XML\_Map** connector as the **Source File**, which maps to the **Source** tree.
  3. Data from the **CSV1** connector passes to the **XML\_Map** connector as the **Destination File**, which maps to the **Destination** tree. ![](/26.4/images/dbsa_xmlmap_select.png)
  4. The **XML\_Map** connector attempts to automatically map **Source** elements to **Destination** elements with the same name. You can manually change these and fill in blank mappings by dragging elements from **Source** to **Destination**. ![](/26.4/images/dbsa_xmlmap_elements.png)
  5. The file from **CSV1** that has been modified with data from **Shopify\_Select** then passes down the flow from **CSV1**.
# Shopify Upsert Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/shopify/shopify-upsert Insert or update Shopify data from an Arc message in the flow. The Upsert action updates or inserts Shopify data. By default, if a record already exists in Shopify, an update is performed on the existing data in Shopify using the values provided from the input. See [UPSERT Query Behavior](#upsert-query-behavior) for more information. ## Tables and Columns The Shopify Upsert connector supports two configuration modes: **Table Structure** and **Mapping**. **Table Structure** is the default. When the **Mapping** toggle is off, click **Add Table Structure** to select a target table from Shopify. ![](/26.4/images/dbsa_upsert_landing_page.png) On the **Edit Destination Structure** pane, select the desired table(s) and click **Apply**. ![](/26.4/images/dbsa_upsert_table_modal.png) > [!NOTE] > > See [Child Tables](/26.4/self-hosted/en/connectors/shopify/shopify-ac#child-tables) for more information about selecting multiple related tables. The selected table(s) and columns appear. The connector automatically detects values that have special relevance, such as the primary key, and labels them accordingly. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_table_structure.png) **Mapping** mode enables a full mapping editor. Enable the **Mapping** toggle to switch to this mode. ![](/26.4/images/dbsa_upsert_mapping_editor.png) In the **Destination** pane, click **Add Destination Structure**, select the target table(s), and click **Apply**. ![](/26.4/images/dbsa_upsert_add_destination_modal.png) The selected table(s) and columns appear in the **Destination** pane. Right-click a column to access additional settings, including UPSERT settings for that column. ![](/26.4/images/dbsa_upsert_destination_structure.png) > [!NOTE] > > A source template is not required to add a destination structure or configure UPSERT settings. You can set the UPSERT key and save in structure-only mode before connecting a source. ## UPSERT Query Behavior UPSERT settings are configured per column in both Table Structure and Mapping modes. Right-click a column and select **UPSERT Settings** from the context menu. The **UPSERT Settings** dialog opens. By default, the **Enable UPSERT settings** toggle is **ON**, meaning the connector checks whether a record already exists in the destination before deciding to INSERT or UPDATE it. ![](/26.4/images/dbsa_upsert_settings_dialog.png) The query used to determine whether a record exists is controlled by the logic described in the following sections. ### UPSERT Settings Enabled When **Enable UPSERT settings** is on, query behavior depends on the option selected in the **UPSERT Settings** list: **UPSERT by** (match on a single column) or **Query to select UPSERT key** (use a custom lookup query). #### UPSERT By Id When you select **UPSERT by** in the **UPSERT Settings** list and choose the table's key column, the connector issues a query to determine if any records exist with the matching key value: ```text theme={null} The action type is UPSERT and an UPSERT query is not specified. The key column [Id] is present and will be determined using an automatically generated UPSERT query. SELECT COUNT(*) AS [count] FROM [Orders] WHERE [Id] = @Id; @Id='123' ``` A match results in an update; no match results in an insert. If the key column is not present in the input XML, the record is always inserted: ```text theme={null} The key column Id was not specified, INSERT will be executed. The action type is UPSERT or an UPSERT query is specified. The key column is not present. Record will be INSERTED. ``` #### UPSERT By Non-key Column When you select **UPSERT by** in the **UPSERT Settings** list and choose a non-key column, the connector issues a query to determine if any records exist where that column contains the matching value, targeting the primary key in the SELECT (the following examples illustrate UPSERTing to an Orders table): ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId='12345' ``` The SELECT query always targets the key column(s) of the table in the SELECT statement. If there is a successful match, the data in the input XML is updated into the data source. If there is no match, the record is inserted instead. > [!NOTE] > > If the selected **UPSERT by** column is not specified in the input, the lookup is used with a NULL value, as shown in the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId; @OrderId=NULL ``` #### Using a Custom Query When you select **Query to select UPSERT key** in the **UPSERT Settings** list, you can write a custom query to determine the UPSERT key. This allows the lookup to use more complex logic than a single column match. Consider the following query: ```sql theme={null} SELECT [Id] FROM [Orders] WHERE [OrderId] = @OrderId AND Status = 'Active' ``` In this example, you must select the key column(s) of the table, to provide adequate criteria for the UPDATE. The `@OrderID` syntax is used to reference a value from the matching input XML. You can also use static values: the quotes around `'Active'` denote that this clause uses a static value. Again, a successful match of the key column is used to form an update request. If there is no match, the record is inserted. > [!TIP] > > When in doubt about the behavior of a database insertion, review the `.log` file for the UPSERT attempt: a complete breakdown of the logic used to guide the behavior is sent to the log file. ### UPSERT Settings Disabled To disable UPSERT behavior for a column, open the **UPSERT Settings** dialog (right-click the destination column and select **UPSERT Settings**), toggle **Enable UPSERT settings** to **OFF**, and click **Apply**. When UPSERT settings are disabled, the connector issues only INSERT queries. Examining the code view of the input mapping shows that the `action` attribute of the table is `insert`: ```xml theme={null} ``` All queries issued to the table are treated as inserts, whether or not the primary key is present. To have all received records updated against the table, you can set this action keyword to `update`, as shown in the following snippet: ```xml theme={null} ``` > [!WARNING] > > When the action is defined as `update`, the application expects the key column of the table (for example, Id) to be present. If the key column is not provided, the effects of the UPDATE command can vary by target database type, so you might see more rows changed than expected, or that none are changed at all. ## Example Arc Flow After you choose the tables and columns for the Upsert-configured Shopify connector, place the connector at the end of an Arc flow. The screenshot below depicts an example flow with a File connector, a CSV connector, and a Shopify Upsert connector at the end: ![](/26.4/images/shopify_upsert.png) Data in this example flow passes through these steps:
  1. The File connector pulls a CSV file from disk and into the flow.
  2. The **CSV1** connector translates the CSV file into XML.
  3. The **Shopify\_Upsert** connector receives the XML data. Configure column mapping directly in the connector on the **Action** tab using Mapping mode, as described in [Tables and Columns](#tables-and-columns). The connector then performs the Upsert and passes the resulting data to Shopify.
# Snowflake Connector Setup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake Connect Arc to Snowflake for data integration and synchronization The Snowflake connector allows you to integrate Snowflake into your data flow by pushing or pulling data from Snowflake. Watch this short video for an overview of how you can integrate Snowflake into your B2B automations using Arc. [https://www.youtube.com/embed/52aLvW9QTGw](https://www.youtube.com/embed/52aLvW9QTGw) ## Key Capabilities * Cloud data warehouse platform integration with secure data loading, querying, and Snowflake-specific authentication * Standard database operations (Select, Lookup, Upsert, and Stored Procedure) for data manipulation Follow the steps below to connect Arc to Snowflake. ## Establish a Connection To allow Arc to use data from Snowflake, you must first establish a connection to Snowflake. There are two ways to establish this connection: * Add a Snowflake connector to your flow. Then, in the **Settings** tab, click **Create** next to the Connection drop-down list. * Open the Arc **Settings** page, then open the **Connections** tab. Click **Add**, select **Snowflake**, and click **Next**. > [!NOTE] > > * The login process is only required the first time the connection is created. > * Connections to Snowflake can be reused across multiple Snowflake connectors. ### Enter Connection Settings After opening a new connection dialogue, follow these steps: 1. Provide the requested information: * **Name** — The static name of the connection. * **Type** — This is always set to Snowflake. * **Connection** — Choose your authentication scheme from the list. Options are **Password**, **OAuth**, **Okta**, **PrivateKey**, **Azure AD**, or **PingFederate**. * **URL** — The full URL of your Snowflake database (Password and OAuth authentication only). * **User** — The username for authentication with the Snowflake database (all authentication schemes except OAuth). * **Password** — The password for the authenticating user (Password and Okta authentication only). * **OAuth Client Id** — The client Id for your account (OAuth authentication only). * **OAuth Client Secret** — The client secret for your account (OAuth authentication only). * **Private Key** — The private key for your account (PrivateKey authentication only). * **Private Key Type** — The private key type for your account (PrivateKey authentication only). * **Private Key Password** — The password for your private key (PrivateKey authentication only). * **Warehouse** — The name of the Snowflake warehouse. * **Schema** — The schema of the Snowflake database. * **Database** — The name of the Snowflake database. 2. Optionally, click **Advanced** to open the drop-down menu of advanced connection settings. In most cases, you should not need these. 3. Click **Test Connection** to ensure that Arc can connect to Snowflake with the provided information. If an error occurs, check all fields and try again. 4. Click **Add Connection** to finalize the connection. 5. In the **Connection** drop-down list of the connector configuration pane, select the newly-created connection. 6. Click **Save Changes**. > [!NOTE] > > There are also data source-specific authentication and configuration options on the **Advanced** tab. They are not all described in this documentation, but you can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. ## Select an Action After establishing a connection to Snowflake, you must choose the action that the Snowflake connector will perform. The table below outlines each action and where it belongs in an Arc flow. | Action | Description | Position in Flow | | - | - | - | | [Upsert](./snowflake-upsert) | Inserts or updates Snowflake data. By default, if a record already exists in Snowflake, an update is performed on the existing data in Snowflake using the values provided from the input. | End | | [Lookup](./snowflake-lookup) | Retrieves values from Snowflake and inserts those values into an already-existing Arc message in the flow. The **Lookup Query** determines what value the connector will retrieve from Snowflake. It should be formatted as a SQL query against the Snowflake tables. | Middle | | [Lookup Stored Procedure](./snowflake-lsp) | Treats data coming into the connector as input for a stored procedure, then inserts the result into an existing Arc message in the flow. You can click the **Show Sample Data** button in the **Test Lookup** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | | [Select](./snowflake-select) | Retrieves data from Snowflake and brings it into Arc. You can use the **Filter** panel to add filters to the Select. These filters function similarly to *WHERE* clauses in SQL. | Beginning | | [Execute Stored Procedures](./snowflake-esp) | Treats data coming into the connector as input for a stored procedure, then passes the result down the flow. You can click the **Show Sample Data** button in the **Test Execute Stored Procedure** modal to provide sample inputs to the selected Stored Procedure and preview the results. | Middle | ## Advanced Tab Many of the settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. You can find detailed information for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. The options described below are available for all data sources. ### Message > [!NOTE] > > The **Save to Sent Folder** and **Sent Folder Scheme** settings are deprecated and hidden by default. They appear only on connectors where they were previously enabled or configured to a non-default value. To retain copies of successfully processed files, right-click the connector in the flow, choose **Show Success Path**, and connect the success path to a [File connector](../file). | Setting | Description | | - | - | | **Save to Sent Folder** (Deprecated) | Copies files processed by the connector to the Sent folder. Disabled by default. Files saved to the Sent folder are not covered by Encryption at Rest. | | **Sent Folder Scheme** (Deprecated) | Groups files in the Sent folder by the selected time interval. Only relevant when **Save to Sent Folder** is enabled. | ### AI
Setting Description
**Customize AI Prompts** Only visible when **Mapping** is enabled on the **Action** tab. Check this to override the default System Prompt and User Prompt content that the AI-assisted mapping uses to create the mappings between the source and destination documents. When unchecked, the connector uses the default prompts to generate the mappings. The default prompts are shown in the text boxes when the checkbox is enabled for the first time.
![Customize AI Prompts settings](/26.4/images/customize_ai_prompts.png) ### Advanced Settings *Advanced settings* are common settings for connectors that rely on arbitrary database drivers (for example, Database, CData, and API connectors) in order to make connections to various data sources. | Setting | Description | | - | - | | **Auto Truncate** | When enabled, the connector automatically truncates any string or binary column values that are longer than the allowed limit. | | **Command Timeout** | The command execution timeout duration in seconds. | | **Last Inserted Id Query** | Provide a query to execute to retrieve the auto-increased Id for the last inserted record. | | **Log Level** | The verbosity of logs generated by the connector. When you request support, set this to **Debug**. | | **Local File Scheme** | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see [Macros](#macros). | | **Log Messages** | When checked, the connector keeps a copy of the message next to the logs for the message in the Logs directory. If you disable this, you might not be able to download a copy of the file from the **Input** or **Output** tabs. | | **Max Failed Records** | The maximum number of records that are allowed to fail during insertion to allow processing to continue. The default value of 0 means that any errors cause the input message to be marked as an Error, and any uncommitted transactions are rolled back. A value of -1 means that all errors are ignored and the connector continues attempting to insert subsequent records. A positive value means that the connector continues attempting to insert records until the threshold is reached. | | **Output File Format** | The format in which output data is represented. The default value (XML) causes the connector to output an XML file for every record that is processed, while optionally combining multiple records into a single data structure (depending on the value of **Max Records**). The CSV and TSV options output data in their respective file formats. This option is not available for complex table structures that include child tables. | | **Process Changes Interval Unit** | Applicable to Select action only. When **Use column *columnname* for processing new or changed records** is checked in the **Advanced** portion of the **Select Configuration** section of the **Settings** tab, this controls how to interpret the **Process Changes Interval** setting (for example, Hours, Days, or Weeks). See [Only Process New or Changed Records](./snowflake-ac#only-process-new-or-changed-records) for details on this and the following two settings. | | **Process Changes Interval** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls how much historical data Arc attempts to process on the first attempt. For example, keeping the default of 180 (Days) means Arc only attempts to process data that has been created or modified in the last 180 days. | | **Process Changes Start Date** | When **Use column *columnname* for processing new or changed records** is checked on the **Select Configuration** section of the **Settings** tab, this controls the start date of the data to retrieve. | | **Reset History** | Resets the cache stored when **Use column *columnname* for processing new or changed records** is checked. | | **Batch Input Size** | The maximum number of queries in a batch if batches are supported by the data source. | | **Batch Size** | The maximum number of batch messages in a batch group. | | **Max Records** | The maximum number of records to include in a single output message. Use -1 to indicate that all output records should be put in a single file, and use 0 to indicate that the connector can decide based on the configured **Output File Format**. By default, XML outputs one record per file, and flat file formats include all records in one file. | | **Transaction Size** | The maximum number of queries in a transaction. | | **Processing Delay** | The amount of time (in seconds) by which the processing of files placed in the Input folder is delayed. This is a legacy setting. Best practice is to [use a File connector](/26.4/self-hosted/en/flows/designing-a-flow#interacting-with-the-local-file-system) to manage local file systems instead of this setting. | | **Insert Empty** | When left unchecked (the default), the connector skips empty XML elements when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes empty XML elements in INSERT and UPDATE statements, using an empty string as the value for the column. | | **Insert Null** | When left unchecked (the default), the connector skips null XML elements (indicated by the `xsi:nil="true"` attribute) when building INSERT and UPDATE statements. This causes the database to use the default column value for INSERT statements, and for column values to remain unchanged for UPDATE statements. When checked, the connector includes null XML elements in INSERT and UPDATE statements, using null as the value for the column. The cases which are treated as null are: `Not exist`, ``, ``, ``, ``, and ``. | | **Lookup Caching** | Check **Enable lookup cache** to perform queries against a local cache before they are executed directly against the data source (when no records are returned from the cache). When left unchecked (the default), queries are performed directly against the data source. The following lookup cache options are only available when **Enable lookup cache** is checked. | | **Lookup Cache Provider** | Specify which driver to use to facilitate the connection to the cache database. | | **Lookup Cache Connection String** | When set, the connector uses the database specified here for caching. | | **Lookup Cache Tolerance** | Specify how often the connector should query the data source to refresh the cache. Used in conjunction with the **Tolerance Unit** option. | | **Lookup Cache Tolerance Unit** | Select **Minute**, **Hour**, or **Day**. | | **Log Subfolder Scheme** | Instructs the connector to group files in the `Logs` folder according to the selected interval. For example, the **Weekly** option instructs the connector to create a new subfolder each week and store all logs for the week in that folder. The blank setting tells the connector to save all logs directly in the `Logs` folder. For connectors that process many transactions, using subfolders can help keep logs organized and improve performance. | ### Miscellaneous *Miscellaneous settings* are for specific use cases. | Setting | Description | | - | - | | **Other Settings** | Enables you to configure hidden connector settings in a semicolon-separated list (for example, `setting1=value1;setting2=value2`). Normal connector use cases and functionality should not require the use of these settings. | ## Automation Tab ### Automation Settings *Settings related to the automatic processing of files by the connector.* | Setting | Description | | - | - | | **Send** | Whether files arriving at the connector are automatically sent. | | **Retry Interval** | The number of minutes before a failed send is retried. | | **Max Attempts** | The maximum number of times the connector processes the file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. | | **Receive** | Whether the connector should automatically query the data source. | | **Interval** | The interval between automatic query attempts. | | **Minutes Past the Hour** | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to *Hourly*. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. | | **Time** | The time of day that the attempt should occur. Only applicable when the interval setting above is set to *Daily*, *Weekly*, or *Monthly*. | | **Day** | The day on which the attempt should occur. Only applicable when the interval setting above is set to *Weekly* or *Monthly*. | | **Minutes** | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to *Minute*. | | **Cron Expression** | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to *Advanced*. | | **Start Date and Time** | The date and time at which the automation first runs. Every run after that follows the interval setting above. The last and next run times appear in the [Automation Status](#automation-status) section. | ### Automation Status *A read-only summary of whether the automation is running, what happened on the last run, and when the next run occurs.* The **Automation Status** card appears under the **Enabled** checkbox on the **Automation** tab for **Receive**, **Download**, and **Scheduled Send** automation. The card answers three questions: whether the automation ran, whether the run worked, and when the next run occurs. The card updates as you change the **Enabled** checkbox, or the **Interval** or **Start Date and Time** settings, before you save the connector. ![The Automation Status card on the Automation tab showing a Suspended status, a Failed last processed result, and the next run time](/26.4/images/automation-status-card.png) Status The current state of the automation for the connector.
Status Description
**In Progress** A run is executing now, or another instance has locked the scheduled slot.
**Idle** The automation is waiting for the next scheduled run.
**Suspended** The connector is in a [backoff](/26.4/self-hosted/en/connectors/connector-backoff) state after repeated failed runs, and the automation service is skipping it.
**Disabled** The automation is turned off for the connector.
Last Run Time The outcome of the most recent run.
Field Description
**Date** The scheduled time of the last run.
**Last Processed Start** The time the connector actually started processing the last run. This can be slightly later than the scheduled time.
**Last Processed Result** The outcome of the last run:
  • **Successful**: The run completed and processed data.
  • **No Data**: The run completed but there was nothing to process, such as a query that returned no rows.
  • **Failed**: The run did not complete.
**Last Successful At** The date and time of the most recent successful run. This field appears when the last run failed or returned no data, so you can see how long the connector has gone without a successful run.
Next Run Time The **Date** field shows when the automation is scheduled to run next. It is blank when the automation is disabled. View Details The **Suspended** status, the **Failed** result, and the **No Data** result each include a **View Details** link that opens a dialog with the full message:
  • **Suspended**: Lists each instance that is in backoff, how many times it failed, and when the automation service resumes. See [Connector Backoff](/26.4/self-hosted/en/connectors/connector-backoff) for how a connector exits this state.
  • **Failed**: Shows the error message from the last failed run.
  • **No Data**: Shows the message the connector reported, such as `0 rows returned`.
No Run Versus No Data The card distinguishes a connector that has not run from one that ran and found nothing. If **Last Processed Result** is empty, the automation has not run yet, so check the **Next Run Time** and the schedule settings. If the result is **No Data**, the automation ran on schedule and the data source returned nothing, so check the source for new data. Disabled Connectors A disabled connector keeps its last run information, so you can still see when it last ran and what happened. Only the **Next Run Time** is cleared. ### Performance *Settings related to the allocation of resources to the connector.* | Setting | Description | | - | - | | **Max Workers** | The maximum number of worker threads consumed from the threadpool to process files on this connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | | **Max Files** | The maximum number of files sent by each thread assigned to the connector. If set, this overrides the default setting on the [Performance Settings](/26.4/self-hosted/en/getting-started/administration/settings/performance-settings) portion of the [Advanced Settings](/26.4/self-hosted/en/getting-started/administration/settings/advanced-settings) page. | ## Alerts Tab *Settings related to configuring alerts.* Before you can execute Service Level Agreements (SLAs), you need to set up email alerts for notifications. By default, Arc uses the global settings on the [Alerts](/26.4/self-hosted/en/getting-started/administration/settings/alerts) tab. To use other settings for this connector, toggle **Override global setting** on. By default, error alerts are enabled, which means that emails are sent whenever there is an error. To turn them off, uncheck the **Enable** checkbox. Enter a **Subject** (mandatory), then optionally enter a comma-separated list of **Recipient** emails. ## SLAs Tab *Settings related to configuring Service Level Agreements (SLAs).* SLAs enable you to configure the volume you expect connectors in your flow to send or receive, and to set the time frame in which you expect that volume to be met. Arc sends emails to warn the user when an SLA is not met, and marks the SLA as *At Risk*, which means that if the SLA is not met soon, it will be marked as *Violated*. This gives the user an opportunity to step in and determine the reasons the SLA is not being met, and to take appropriate actions. If the SLA is still not met at the end of the at-risk time period, the SLA is marked as violated, and the user is notified again. To define an SLA, toggle **Expected Volume** on, then click the **Settings** tab. ![Empty SLA configuration](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_empty.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=45e386d064d0de84861eae628ed1f166) * If your connector has separate send and receive actions, use the radio buttons to specify which direction the SLA pertains to. * In the **Expect at least** portion of the window: * Set the minimum number of transactions you expect to be processed (the volume) * Use the **Every** fields to specify the time frame * Indicate when the SLA should go into effect. If you choose **Starting on**, complete the date and time fields. * Check the boxes for the days of the week that you want the SLA to be in effect. Use the dropdown to choose **Everyday** if necessary. * In the **Set status to 'At Risk'** portion of the window, specify when the SLA should be marked as at risk. * By default, notifications are not sent until an SLA is in violation. To change that, check **Send an 'At Risk' notification**. The following example shows an SLA configured for a connector that expects to receive 1000 files every day Monday-Friday. An at-risk notification is sent 1 hour before the end of the time period if the 1000 files have not been received. ![SLA Configuration Example](https://mintcdn.com/cdata-arc/JuVWSaGlKX7avH_C/26.4/images/sla_defined.png?fit=max\&auto=format\&n=JuVWSaGlKX7avH_C\&q=85\&s=50424ba4046c10f47f4cb8a7b31892d1) > [!NOTE] > > You can turn off SLA alerts if necessary. This can be useful during maintenance windows. Click **Settings** on the navbar, then navigate to **Alerts > General Alerts**. Click the tablet and pencil icon to edit, and uncheck the **SLA Alerts** setting. ## Macros Using macros in file naming strategies can enhance organizational efficiency and contextual understanding of data. By incorporating macros into filenames, you can dynamically include relevant information such as identifiers, timestamps, and header information, providing valuable context to each file. This helps ensure that filenames reflect details important to your organization. Arc supports these macros, which all use the following syntax: `%Macro%`. | Macro | Description | | - | - | | ConnectionName | Evaluates to the connection name used by the connector. | | ConnectorID | Evaluates to the ConnectorID of the connector. | | Ext | Evaluates to the file extension of the file currently being processed by the connector. | | Filename | Evaluates to the filename (extension included) of the file currently being processed by the connector. | | FilenameNoExt | Evaluates to the filename (without the extension) of the file currently being processed by the connector. | | MessageId | Evaluates to the MessageId of the message being output by the connector. | | RegexFilename:*pattern* | Applies a RegEx pattern to the filename of the file currently being processed by the connector. | | Header:*headername* | Evaluates to the value of a targeted header (`headername`) on the current message being processed by the connector. | | LongDate | Evaluates to the current datetime of the system in long-handed format (for example, Wednesday, January 24, 2024). | | ShortDate | Evaluates to the current datetime of the system in a yyyy-MM-dd format (for example, 2024-01-24). | | DateFormat:*format* | Evaluates to the current datetime of the system in the specified format (`format`). See [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#date-formats-with-literal-characters) for the available datetime formats | | Vault:*vaultitem* | Evaluates to the value of the specified vault item. | ### Examples Some macros, such as %Ext% and %ShortDate%, do not require an argument, but others do. All macros that take an argument use the following syntax: `%Macro:argument%` Here are some examples of the macros that take an argument: * %Header:headername%: Where `headername` is the name of a header on a message. * %Header:mycustomheader% resolves to the value of the `mycustomheader` header set on the input message. * %Header:ponum% resolves to the value of the `ponum` header set on the input message. * %RegexFilename:pattern%: Where `pattern` is a regex pattern. For example, `%RegexFilename:^([\w][A-Za-z]+)%` matches and resolves to the first word in the filename and is case insensitive (`test_file.xml` resolves to `test`). * %Vault:vaultitem%: Where `vaultitem` is the name of an item in the [vault](/26.4/self-hosted/en/getting-started/administration/settings/global-settings-vault). For example, `%Vault:companyname%` resolves to the value of the `companyname` item stored in the vault. * %DateFormat:format%: Where `format` is an accepted date format (see [Sample Date Formats](/26.4/self-hosted/en/scripting/value-formatters/date-formatters#sample-date-formats) for details). For example, `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` resolves to the date and timestamp on the file. You can also create more sophisticated macros, as shown in the following examples: * Combining multiple macros in one filename: `%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%%EXT%` * Including text outside of the macro: `MyFile_%DateFormat:yyyy-MM-dd-HH-mm-ss-fff%` * Including text within the macro: `%DateFormat:'DateProcessed-'yyyy-MM-dd_'TimeProcessed-'HH-mm-ss%` # Snowflake Advanced Configuration Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake-ac Advanced configuration options for the Snowflake connector in CData Arc. The **Advanced** tab allows you to configure advanced properties of the Snowflake connector. The sections below detail the functionality and configuration of:
  • [XML models](#xml-models)
  • [Child tables](#child-tables)
  • [Advanced record selection](#only-process-new-or-changed-records)
  • [Pre- and Post-processing queries](#pre-processing-and-post-processing-queries)
  • [XML attributes](#xml-attributes)
Many of the other settings on the **Advanced** tab are dynamically loaded from the data source you are working with, so they are not covered in this documentation. Settings that are not data source-specific are described in [Advanced Tab](#advanced-tab). You can find detailed information on *all* configuration settings for your data source on the [Online Help Files](https://www.cdata.com/kb/help/) page of the CData website. From this page, click the tile for your data source. > [!TIP] > > You can highlight any field in the application to display a tooltip with more information about the functionality of that field. ## XML Models The Snowflake connector models Snowflake tables in CData Arc as XML. The modelling follows this structure: * A parent element identifies the table to insert into. * Each child element corresponds to a column in the target table. This establishes a connection between XML elements and Snowflake values, which allows the connector to read values from incoming XML documents and use them in INSERT or UPDATE queries. The XML Model for a table can be viewed by clicking **`` Code** above the configuration pane. The following XML is a simplified example model for a table called **Account**, which is modeled as the XML element ``: ```xml theme={null} ``` In the above example, `Id`, `account_c`, `AccountNumber`, and `AccountSource` are columns in the **Account** table. When the connector processes an XML file that matches this structure, it automatically inserts the values from that XML into the **Account** table. When the connector queries the **Account** table, it outputs XML that matches this structure. ### Batching Input The Snowflake connector supports batching to improve performance when inserting large sets of data. Batching is configured via two fields under the **Advanced Settings** section: **TransactionSize** and **BatchInputSize**. When these fields are set to positive integers, the connector inserts \[*BatchInputSize*] records in a single operation (a batch), and it inserts \[*TransactionSize*] records in a single connection (a transaction). An XML input file can contain more records than a single transaction size. If this happens, the connector inserts the data over multiple transactions. If an error occurs during a batch insert, the connector rolls back the transaction and caches the index of the first record in the failed transaction. When the input file is retried, only records after the cached index are inserted into Snowflake. ### Batching Output When querying for output, you can configure the connector to support batching records together into a single document. The **Max Records** setting governs how many records should be included in a single message, and the **Batch Size** setting indicates how many Batch Messages should be included in a single Batch Group. ## Child Tables The Snowflake connector supports Upserting and Selecting multiple tables with a single action. When you create a table model with multiple tables, the tables should have a foreign key relationship where one table is the child of another table. To interface with child tables, create a model that targets the parent table, then use the arrow on the right side to add child tables. The following image shows an Upsert action. ![](/26.4/images/dbsa_upsert_table_modal.png) When you click **Apply**, the designer populates the hierarchical structure of tables. Select which columns of the child table to include in the model. ### Child Table XML The Input Mapping XML structure is multi-layered when inserting into parent and child tables. The following XML snippet is an example Input Mapping for two tables related by a foreign key: ```xml theme={null} ``` In the above example, `Account` is the parent table and `AccountPartner` is the child table. When the Snowflake connector processes an input XML file that matches this structure, it updates both tables and ensures that the keys linking the two records are the same. ### Using the ref Attribute Some columns in child tables are marked with a [ref](#ref) tag. This tag indicates that these columns correspond to the primary key in the parent table. After making the insert into the parent table, and before making the insert into the child table, the connector queries the parent table for the primary key of the record it just inserted. The connector then uses this value when inserting into the child table. This is useful when the primary key of the parent table is generated by Snowflake. In this case, this column is not passed in as input XML, so the connector does not know the value of the column until after it has inserted the record into the parent table. In the XML model, the `ref` attribute indicates that a column can be referenced later or that the value for a column should be read from another column in the mapping. In order to reference the value of columns that are generated by Snowflake, the column element should include a `ref` attribute set to the reserved value `LAST_INSERT_ID`: ```xml theme={null} ``` This indicates that the *id* value will be retrieved after the insert and can be referenced later (usually in a child table). To reference this value later, set the `ref` element to the name of the relevant column: ```xml theme={null} ``` In the above example, *id* is the name of the primary key in the parent table, and it is also the name of a column in the child table. The `ref` syntax instructs the connector to retrieve the *id* from the parent table after inserting, and then use that retrieved value as the *id* column in the child table. ### Selecting From Child Tables Typically, when Selecting from a parent table and a child table, records should only be pulled from the child table if a foreign key matches the primary key from the parent table. For example, imagine a Select that pulls purchase order data from a **PurchaseOrders** parent table and a **PurchaseOrderLineItems** child table. Each record (order) pulled from **PurchaseOrders** should also pull the line items associated with that order. The two tables are related by a shared *PONumber* column which is the primary key of the **PurchaseOrders** table and a foreign key in the **PurchaseOrdersLineItems** table. Line item records should only be pulled from the child table if the line item's *PONumber* matches the *PONumber* of the order being pulled from the parent table. To establish this relationship in Arc, add a new Filter to the Select and configure it as follows:
  • In the left-hand dropdown, select the column of the child table that is the foreign key to the parent table. In the above example, this is the *PONumber* column of the **PurchaseOrderLineItems** table.
  • In the middle (operator) dropdown, select **equal**.
  • In the right-hand input box, first use the dropdown arrow to the right to select **REF**. Afterwards, the input box should have the `$` prefix, indicating that it is referencing the parent table.
  • Use the right-hand dropdown to select the parent table column that should match the child table's foreign key. In the above example, this is the *PONumber* column of the **PurchaseOrders** table.
## Only Process New or Changed Records Select actions can be configured to only retrieve records that have been updated or added recently. The connector can use two approaches for detecting which records are new or modified.
  • The first approach requires a DateTime column that represents the last-modified time for records.
  • The second approach requires a column that explicitly stores a value representing whether a record should be processed or not.
Both options are set on the **Settings** tab in the **Advanced** settings at the bottom of the configuration panel. To further refine your requirements, see the three additional **Process Changes...** settings in the [Advanced Settings](#advanced-settings) portion of the **Advanced** tab. ### Using a Last Modified Column If the table includes a DateTime column, a checkbox appears in the **Advanced** section to **Use column for processing new or changed records**. Set this to the DateTime column that represents the last-modified time for records. ### Using a Custom Column Select actions can be configured to update a column in the table for records that are successfully pulled from Snowflake. You can combine this with a Filter rule that only selects records with a column value that indicates they have not yet been pulled (or should be pulled again). To enable column updating after processing, use the **Update column with value when rows are processed successfully** checkbox in the **Advanced** section, then specify which column should be updated, and what value should be set in the column. For example, you could configure an Output Mapping to update the *Processed* column to **1** after pulling a record. Then you could apply a **Filter** rule that restricts records to only those where *Processed not equal 1*. In this example, when the connector processes the Select template, it creates a SELECT statement based on the template. The template executes `SELECT Id, AccountId, Amount FROM INVOICE`, creates an XML file for every record in the result set based on the template, and places those files in the Output folder for this connector. ## Pre-Processing and Post-Processing Queries You can configure Snowflake connectors to execute additional queries before and after executing their configured actions. To configure these additional queries you need to manually modify the query code that the connector generates. The sections below outline this process. ### In Upsert Mappings In an Upsert mapping, it is possible to issue queries that automatically take place before or after all records are upserted into a table. To configure this feature, first click **`` Code** in the **Upsert Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_upsertcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=1c4b5205d04d91825a7ef3f257ef8cee) This opens a text editor that allows you to edit advanced settings for a table mapping. The text in the editor might resemble this snippet: ```xml theme={null} … ``` To issue a query before or after any columns are added to the table, define an element with the following attributes: * `requireInput`: Set to `false`. This attribute value instructs the connector to issue the query even without any matching input element in the input file. * `insertQuery`: Contains the query that you want to execute either before or after the UPSERT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is updateTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the UPSERT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo.FlushTemp]` before any UPSERT requests to the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[CommitTemp]` after all UPSERT requests to the table are completed. ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in an Upsert mapping do not accept input elements that are not hardcoded in the query, and they do not generate output data. ### In Select Mappings In a Select mapping, it is possible to issue queries that automatically take place before or after queries are issued to the targeted table. To configure this feature, first click **`` Code** in the **Select Configuration** pane. ![](https://mintcdn.com/cdata-arc/mPTGrzt-QAW42BUU/26.4/images/dbsa_ac_selectcode.png?fit=max\&auto=format\&n=mPTGrzt-QAW42BUU\&q=85\&s=b6d9b3690a54ab4e408d4f4f7bbd01b5) This opens a text editor that allows you to edit advanced settings for a table mapping. To issue a query before or after any columns are added to the table, define an element with the following attributes: * `outputResult`: Set to `false`. This attribute value instructs the connector not to return any data in the output file as a result of the query. * `selectQuery`: Contains the query that you want to execute either before or after the SELECT query. * `table` (optional): Set to the name of the table that you want to target with the extra query. (In the examples below, the table name is selectTable.) If you issue the query at the root level of the mapping—that is, as a child of the `` element—this attribute is not required. If you issue a query as a child to another table, setting the table attribute ensures that the query is distinguished from a column of that table. Place the newly defined element before or after the SELECT query, depending on when you want it to execute. In this example, the element `` executes the query `EXEC [dbo].[prepTable]` before the SELECT query is issued against the table: ```xml theme={null} … ``` In this example, the element `` executes the query `EXEC [dbo].[cleanTable]` after the SELECT query and any child queries are issued against the table: ```xml theme={null} … ``` The element names `runFirst` and `runLast` are placeholders. You can modify them to meet your requirements. > [!NOTE] > > PreProcessing and PostProcessing queries in a Select mapping do not accept input elements that are not hardcoded in the query. ## XML Attributes Arc supports a set of XML attributes that define how data is interpreted, mapped, and transformed between XML documents and database tables. These attributes can be applied at different levels—such as the overall XML structure, specific tables, or individual columns—to precisely control how Arc reads, writes, and formats database data. ### Table Attributes Table attributes define how Arc interacts with database tables, specifying which data to retrieve, how to insert or update records, and whether to process only changed rows. Attributes like [selectQuery](#selectquery), [insertQuery](#insertquery), [upsertQuery](#upsertquery), and [table](#table) control data retrieval and modification operations, while settings such as [requireInput](#requireinput), [outputResult](#outputresult), and [ProcessChangesOnly](#processchangesonly) determine input requirements and output behavior. #### allColumns In a Select mapping, use this attribute to specify that the output XML should include all columns returned in the query results. This allows you to define an output mapping without needing to explicitly define the names of all columns in the output mapping. Without this attribute, the columns present in the output mapping are used to position returned elements from the query in the order specified in the code view, and elements that are returned in the query but do not contain a matching column are not returned in the output XML. `` #### insertQuery In an Upsert mapping, you can use this query to override the default query created by the connector when the record is being inserted. It can be used to issue queries beyond the normal Inserts that are associated with an input file. `` #### outputResult In a Select mapping, if you set this attribute, the query matching the Select query is issued without generating content that is to be returned in the output XML. This allows you to run additional queries as part of a transaction that are not reflected in the output of the connector. `` #### processChangesOnly In a Select mapping, if you apply the processChangesOnly attribute to the table in combination with applying the [timeCheck](#timecheck) attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### requireInput In an Upsert mapping, if you set this attribute, the query matching the inputQuery is executed independently of whether the input XML contains the matching element. This is used when a [pre-processing or post-processing query](#pre-processing-and-post-processing-queries) is issued as part of the same transaction where one or more input elements is present (for example, to truncate a table before entering a new batch, or to run a stored procedure after a batch is complete). `` #### selectQuery In a Select mapping, this is used to define the query and its parameters in the XML. If a a selectQuery is not supplied, a simple `SELECT *` is issued. You can use selectQuery for custom queries, such as queries using JOIN or other SQL keywords. `` #### table Used to specify the table that is the target of the mapped query, especially if the table name differs from the valid XML element that maps to it. > [!NOTE] > > If an explicit query is written in selectQuery, upsertQuery, or insertQuery, this attribute is ignored. `` #### upsertQuery In an Upsert mapping, use this query to determine whether the record being processed is a new record (INSERT) or an existing record being updated (UPDATE). The action is determined based on whether a matching result to this query is found. `` ### Column Attributes Column attributes configure how individual data fields are processed, transformed, and validated within your Arc workflows. These attributes allow you to define data types, aggregate values, track changes over time, and apply custom update logic to specific columns. #### aggregate In some cases, the value of a column in a mapping might itself be an XML aggregate. The default behavior is to treat the column as a string and use XML encoding to encode the value of the aggregate in the column. However, when the aggregate attribute is present on a column, the XML file that maps to the table has the value of any aggregate column as raw XML instead of encoded to a single element. `` #### column The column attribute on a column in a mapping provides the name of the column in the table being referenced. This allows you to accomplish several things:
  • Map to columns that are not valid XML elements
  • Create an alias to a column in the database to distinguish it from a similarly named column in another table
  • Use an easier-to-read element in the XML model
`` #### id In a Select mapping, this tags the value of a column with an Id marker that can be referenced later in a child query using the syntax `${#ID}`. This is useful when a column value is needed in a child query where matching column names in multiple tables can create ambiguity in the selection. ```xml theme={null} ... ... ... ``` #### key Identifies the column as a primary key on the table. When this attribute is present on an Upsert query, the connector checks if the record exists before updating it. If it is not present on a table, only Inserts are allowed to the table, and you cannot update column values after selecting records, because there is no way to identify unique records. (You can overwrite an existing column to act as a key if you are positive that it can be used in the same way as a key column.) `` #### ref In an Upsert mapping, this attribute is used to provide values to upsert into the database that cannot be determined from the input XML alone. This is most commonly used in parent-child relationships to determine the database-generated primary key, or for one table to use a foreign key reference on a child table. For more information, see [Using the ref Attribute](#using-the-ref-attribute). ```xml theme={null} ... ... ``` #### timeCheck In a Select mapping, if you apply the [processChangesOnly](#processchangesonly) attribute to the table in combination with applying the timeCheck attribute to a datetime column, the Select mapping uses this column in a dynamically generated WHERE clause in the query to ensure that only new records are retrieved. This corresponds to the column selected in the **Use column *columnname* for processing new or changed records** option in the designer. See [Only Process New or Changed Records](#only-process-new-or-changed-records) for more information. > [!NOTE] > > This is only applicable to the root table in a Select mapping. `` #### type Used to identify the data type of the column in the table. For Upsert queries, the type is validated against the value in the column before the query is executed. `` #### updateValue In a Select mapping, any record selected from the database is updated after retrieval with a new value in this column. This corresponds to the **Update column *columnname* with value *value* when rows are processed successfully** option in the designer. > [!NOTE] > > This is only applicable to the root table in a query. `` # Snowflake Execute Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake-esp Use a stored procedure to process data from an Arc message and pass the result down the flow. The Execute Stored Procedures action treats data coming into the Snowflake connector as input for stored procedures. The connector executes the stored procedure that you choose, then passes the result down the flow (if applicable). Follow these steps to configure a stored procedure:
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc. ![Select stored procedure](/26.4/images/dbsa_esp_select.png) You can see the input and output parameters for the stored procedure, as well as any return values (if applicable).
  2. Click **Test Execute Stored Procedure** to see what the structure of the input XML should look like. ![ESP test](/26.4/images/dbsa_esp_test.png)
  3. Complete the **Input** parameters and click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![ESP sample data](/26.4/images/dbsa_esp_sample.png)
You can connect the Snowflake connector to an XML Map connector. The results of the Execute Stored Procedure appear as inputs under the **Source** tree. ![ESP XML map](/26.4/images/dbsa_esp_xmlmap.png) # Snowflake Lookup Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake-lookup Retrieve values from Snowflake and insert them into an Arc message in the flow. The Lookup action retrieves values from Snowflake and inserts those values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Snowflake connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and helps you build [filter](#filters) conditions.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Snowflake connector to the connector where you want your resulting XML or Header data to go.
## Lookup Configuration Use this section to define the table and columns to include in your query. [Designer Mode](#designer-mode) helps you pick the table and columns you want to target with your query, instead of requiring you to build the query free form. However, you can use [Query Mode](#query-mode) if you prefer to write your query from scratch. ### Designer Mode Follow the steps below to use the Designer to configure a Lookup action for the Snowflake connector. ![Lookup designer](/26.4/images/dbsa_lookup_designer.png)
  1. Use the **Table** dropdown to select the table to include.
  2. Choose the set of **Columns** that you want to include from the table.
  3. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  4. Use **Record Limit** to specify what should happen when the query returns multiple results.
    • If you choose **Use First Result** from the dropdown, you can use the **Error if multiple results** to see an error when multiple results are returned.
    • If you choose **Custom Limit**, you must specify the limit in the field that appears.
  5. Use **Order By** to choose which column to order the results by.
  6. The [Filters](#filters) section lets you create filters to build the query WHERE clause, including references to headers or XML from the input file.
### Filters You can build complex filters by defining individual filters and/or groups of filters. For readability, expand the **Filters** section by clicking the expander icon ![expander](/26.4/images/validate_expander.png) on the right side of the bar. To add a filter, click **Add Filter** or **Add Group**. When you build your filters, use the dropdowns to make your selections and choose your operator. Then specify the value to filter by. You can choose from the following:
  • Message Header
  • Input XML
  • Static Value
When you select **Input XML** for a filter, if you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the image below. ![Lookup filter configuration](/26.4/images/dbsa_lookup_filter.png) The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. This is useful for EDI qualifier-value pairs. For example: `/Interchange/.../REF[REF01='IA']/REF02` targets the REF02 value only when its sibling REF01 equals `IA`. Enter the brackets unescaped here, unlike the backslash-escaped form used inside the *xpath()* formatter. > [!NOTE] > >
  • **Supported forms:** existence checks, equality checks (against child paths, attributes, current text, or quoted literals), and multiple conditions combined with AND. A standalone `[last()]` positional selector is also supported.
  • >
  • **Unsupported forms are rejected**, not silently reinterpreted, so simplify the XPath if Lookup rejects it. Unsupported forms include: arbitrary XPath functions and operators; `[last()]` combined with another condition on the same node; multiple `[last()]` selectors in one path; a numeric selector combined with another condition on the same node; and a condition attached directly to a terminal attribute (rewrite as a parent filter instead, for example `/actor[@type='lead']/@type`).
  • >
  • **Performance:** lookahead syntax causes Lookup to load the document into memory instead of streaming it, which uses more memory. For very large XML input, prefer an XPath with no lookaheads or one that uses a numeric position, or preprocess the input upstream, to keep Lookup in streaming mode.
You can reorder filters and groups. Click the handle ![handle](/26.4/images/xmlmap_condition_handle.png) and drag the object to a new location. ### Query Mode Follow the steps below to use the Query mode to configure a Lookup action for the Snowflake connector. ![Lookup query mode](/26.4/images/dbsa_lookup_query.png)
  1. **Output Destination**: select **XPath** to insert the query response values into an XML file or **Header** to insert the query response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each column in the query has its own XML element (leaf level) in the XML element provided here. The Parent-Level XPath can include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition; uploading a sample file makes it easier to build lookahead paths.
    • Header: When you choose Header, the connector adds one header to the output message for each column in the query. The values are comma-separated for each record that is returned from the database.
  2. In **Query**, enter the SQL query to retrieve data from Snowflake tables.
    • XPath: Format the path as `${XPATH:/path/to/element}`. For example, `SELECT CustomerID FROM Customers WHERE CustomerName = ${XPATH:/Items/Invoices/Customer}`. The XPath can also include [lookahead syntax](/26.4/self-hosted/en/mapping/mapping-node-value-editor#lookahead) to target an element by a sibling condition. For example, `${XPATH:/Interchange/.../REF[REF01='IA']/REF02}` targets the REF02 value only when its sibling REF01 equals `IA`. For supported lookahead forms and performance considerations, see [Filters](#filters).
    • Header: Format the path as `${HEADER:HeaderName}`.
  3. The two checkboxes let you configure what should happen when the query returns multiple results.
    • Check **Use First Result** to always use the first result.
    • Check **Error if multiple results** to see an error when multiple results are returned.
## Advanced ![Lookup advanced settings](/26.4/images/dbsa_lookup_advanced.png)
  1. Use **No Matching Record Found Action** to specify what should happen if no record is found that matches the query.
  2. The Snowflake connector can perform Lookup queries against a local cache instead of querying Snowflake directly for each Lookup. This can improve query performance, especially with larger databases. To enable this, select **Enabled** under **Lookup Caching** and configure the rest of the fields.
    • **Lookup Cache Provider**: Choose which driver to use to facilitate the connection to the cache database.
    • **Lookup Cache Connection String**: When set, the connector uses the database specified here for caching.
    • **Lookup Cache Tolerance**: Controls how often the connector issues a query to the data source to refresh the cache.
    • **Lookup Cache Tolerance Unit**: The **Lookup Cache Tolerance** unit. Choose Minute, Hour, or Day.
    • **Reset Cache**: Clears all cached data and forces the connector to re-query Snowflake on the next Lookup.
> [!NOTE] > >
  • If a query returns no results from the cache, the connector queries Snowflake directly.
  • >
  • If your query statement contains a `limit`, `offset`, or `groupby` parameter, the cache function does not work. These queries go directly to Snowflake.
## Test Lookup Use the **Test Lookup** button to check your configuration and see the result without having to process files manually. Use the **Save Sample Data** button to save the output as a sample for mapping later in the flow. # Snowflake Lookup Stored Procedure Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake-lsp Use a stored procedure to retrieve values from Snowflake and insert them into an Arc message in the flow. The Lookup Stored Procedure action treats data coming into the Snowflake connector as input for stored procedures. The connector executes the selected stored procedure, then inserts the resulting values into an existing Arc message in the flow.
  1. In the CData Arc flow, drag an arrow from the connector that will provide input to the Snowflake connector.
  2. Optionally, choose a **Sample File**. A sample file is recommended because it allows the connector to provide more context about the file structure and makes it easier to select input parameter values.
  3. Once you have completed the configuration steps below, in the Arc flow, drag an arrow from the Snowflake connector to the connector where you want your resulting XML or Header data to go.
## Lookup Stored Procedure Configuration Follow these steps to configure a stored procedure: ![LSP configuration](/26.4/images/dbsa_lsp_config.png)
  1. Select a stored procedure from the drop-down menu. The available stored procedures depend on the connector and cannot be customized from CData Arc.
  2. **Input Parameter**: Add a **Value** for each input parameter. If you have uploaded a sample file, the editor displays a tree representation of the document so you can easily find the XPath you need, as shown in the following image. ![LSP XPath selection](/26.4/images/dbsa_lsp_xpath.png)
  3. **Output Destination**: select **XPath** to insert the stored procedure response value into an XML file or **Header** to insert the stored procedure response value into a message header.
    • XPath: When you choose XPath, the destination is in two parts. The **Parent-Level XPath** should already exist in the XML document. The **New Node Name** is the name of the node that is injected into the XML document to hold the result of the lookup. Each input parameter value has its own XML element (leaf level) in the XML element provided here.
    • Header: When you choose Header, the connector adds one header to the output message for each input parameter. The values are comma-separated for each record that is returned from the database.
  4. **Output Options**: Stored procedure results might include result sets, return values, and/or output parameters. These two checkboxes let you fine-tune what to include in the output.
    • Check **Include Output Parameters** to include the output parameters.
    • Check **Include Result Sets** to include result sets in the output (only applicable when the **Output Destination** is XPath). Because multiple results sets with varying structures might be returned, result sets are injected into the input XML with an additional level of hierarchy.
  5. Click **Test Lookup** to see what the structure of the input XML should look like. ![LSP test](/26.4/images/dbsa_lsp_test.png)
  6. Click **Test** to see what output will be generated. When you are satisfied with the test results, you can click **Save Sample Data** to make the data accessible to other connectors. ![LSP sample data](/26.4/images/dbsa_lsp_sample.png)
# Snowflake Select Source: https://docs.arc.cdata.com/26.4/self-hosted/en/connectors/snowflake/snowflake-select Retrieve data from Snowflake and bring it into Arc. The Select action retrieves data from Snowflake and brings it into CData Arc. ## Tables and Columns Once you have configured and tested your connection, click the **Action** tab. Ensure the **Action** dropdown is set to **Select**. Click **Add Table Structure** to define your tables and columns. Choose a **Structure Type**. The structure type determines how the connector queries Snowflake. The available structure types are:
  • [Parent and Child Tables](#parent-and-child-tables): The connector que