The Google Storage connector uploads to and downloads from the Google Storage cloud storage service.
Key Capabilities
- Google Cloud Storage integration with OAuth authentication and automatic token refresh
- Bucket-based file organization with bidirectional file transfers
- Glob pattern filtering with optional file deletion after download and overwrite protection
- Optional caching to ensure only new or updated files are downloaded
Overview
Each Google Storage connector connects to a specific Google Storage account. Within the remote storage, you set a single bucket as the upload and/or download target. You can specify upload and download paths in the bucket, and apply filters to only send or receive specified file names and/or file extensions.
Files that reach the Google Storage connector in the flow are uploaded to the upload target folder, and files found in the download target folder are downloaded and entered into the flow.
Account authentication uses OAuth, and you must acquire OAuth credentials before connecting with the Google Storage connector. Once authentication is successfully completed with Google, the Google Storage connector handles refreshing the OAuth tokens to ensure that authentication persists over time.
Connector Configuration
This section contains all of the configurable connector properties.
Settings Tab
Authorization
Settings related to authorizing access to the remote storage.
| Setting | Description |
|---|
| Connector Id | The static, unique identifier for the connector. |
| Connector Type | Displays the connector name and a description of what it does. |
| Connector Description | An optional field to provide a free-form description of the connector and its role in the flow. |
| Client Id | The client Id credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. |
| Client Secret | The client secret credential provided by Google for the target storage account. Acquire this value from Google via the Google Console. |
Google might require an OAuth callback/redirect URL when generating OAuth credentials. The callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: src/oauthCallback.rst. For example, if is hosted on ‘mydomain.com’ on port 8001, specify the following URL as the callback/redirect URL: http://mydomain.com:8001/src/oauthCallback.rst.
Host Configuration
Settings related to the target storage host.
| Setting | Description |
|---|
| Bucket Name | The Google Storage bucket to use for uploading and/or downloading files. |
Send
Settings related to uploading to the remote storage.
| Setting | Description |
|---|
| Remote Folder | Files processed by the connector are uploaded to the folder specified here. |
| Overwrite | Whether files that already exist in the remote folder should be overwritten or skipped. |
Receive
Settings related to downloading from the remote storage.
| Setting | Description |
|---|
| Remote Folder | The remote folder from which files are downloaded. Use a comma-delimited list to specify multiple paths. |
| Delete | Check this to delete successfully downloaded files from the remote storage. |
Caching
Settings related to caching and comparing files between multiple downloads.
| Setting | Description |
|---|
| File Size Comparison | Check this to keep a record of downloaded file names and sizes. Previously downloaded files are skipped unless the file size is different than the last download. |
| Timestamp Comparison | Check this to keep a record of downloaded file names and last-modified timestamps. Previously downloaded files are skipped unless the timestamp is different than the last download. |
When you enable caching, the file names are case-insensitive. For example, the connector cannot distinguish between TEST.TXT and test.txt.
Advanced Tab
Advanced Settings
Settings not included in the previous categories.
| Setting | Description |
|---|
| Recurse | Whether to download files in subfolders of the target remote path. |
| Use Virtual Hosting | By default (Auto), the connector tries to determine which request style to use based on the bucket name. Set to True to force the buckets to be referenced using a hosted-style request: http://yourbucket.storage.googleapis.com/yourobject. Set to False to use a path-style request: http://storage.googleapis.com/yourbucket/yourobject. |
| Timeout | The duration the connector waits for a connection response before throwing a timeout error. |
| Receive Filter | A glob pattern filter to determine which files should be downloaded from the remote storage (for example, *.txt). You can use negative patterns to indicate files that should not be downloaded (for example, -*.tmp). Multiple patterns can be separated by commas, with later filters taking priority except when an exact match is found. |
| Local File Scheme | A scheme for assigning filenames to messages that are output by the connector. You can use macros in your filenames dynamically to include information such as identifiers and timestamps. For more information, see Macros. |
| Processing Delay | The amount of time (in seconds) by which the processing of files placed in the Transactions tab is delayed. This is a legacy setting. Best practice is to use a File connector to manage local file systems instead of this setting. |
Proxy Settings
Logging
Miscellaneous
Automation Tab
Automation Settings
Settings related to the automatic processing of files by the connector.
| Setting | Description |
|---|
| Send | Whether files arriving at the connector are automatically uploaded. |
| Retry Interval | The amount of time before a failed upload is retried. |
| Max Attempts | The maximum number of times the connector processes the input file. Success is measured based on a successful server acknowledgement. If you set this to 0, the connector retries the file indefinitely. |
| Receive | Whether the connector should automatically poll the remote download path for files to download. |
| Interval | The interval between automatic download attempts. |
| Minutes Past the Hour | The minutes offset for an hourly schedule. Only applicable when the interval setting above is set to Hourly. For example, if this value is set to 5, the automation service downloads at 1:05, 2:05, 3:05, etc. |
| Time | The time of day that the attempt should occur. Only applicable when the interval setting above is set to Daily, Weekly, or Monthly. |
| Day | The day on which the attempt should occur. Only applicable when the interval setting above is set to Weekly or Monthly. |
| Minutes | The number of minutes to wait before attempting the download. Only applicable when the interval setting above is set to Minute. |
| Cron Expression | A five-position string representing a cron expression that determines when the attempt should occur. Only applicable when the interval setting above is set to Advanced. |
Alerts Tab
SLAs Tab
Establishing a Connection
OAuth credentials are required to establish a connection with the Google Storage connector. Acquire the OAuth credentials from Google via the Google Console, then specify them in the following connector fields on the Settings tab:
- OAuth Client ID
- OAuth Client Secret
Google might require an OAuth callback/redirect URL when generating OAuth credentials. The callback/redirect URL is the same as the address and port where the application is hosted, plus the following resource path: src/oauthCallback.rst. For example, if is hosted on ‘mydomain.com’ on port 8001, specify the following URL as the callback/redirect URL: http://mydomain.com:8001/src/oauthCallback.rst.
Once authentication is successfully completed with Google, the Google Storage connector handles refreshing the OAuth tokens to ensure that authentication persists over time.
Uploading and Downloading
Upload
To upload files, set the Bucket Name field in the Host Configuration portion of the Settings page to the target bucket, then set the Remote Folder field in the Send portion of the Settings page to the folder where files should be uploaded. Each Google Storage connector uploads to a single folder location.
Use the Overwrite remote files option to indicate whether files that already exist in the remote folder should be overwritten or skipped.
After configuration is complete, files placed in the connector Transactions tab are uploaded to the remote storage. If Send Automation is enabled, the upload happens automatically; otherwise, you can send individual files from the Transactions tab.
Download
To download files, set the Remote Folder field in the Receive portion of the Settings page to the folder where files should be downloaded. Use a comma-delimited list to specify multiple folders.
You can set the Receive Filter field on the Advanced tab to a glob filter to specify which files should be downloaded from the remote storage (for example, *.txt). You can use negative patterns to indicate files that should not be downloaded (for example, -*.tmp). Use a comma-separated list to specify multiple filters.
Use Delete to specify whether successfully downloaded files should be removed from the remote storage.
After configuration is complete, files are downloaded according to the Receive Automation settings, or you can manually click the Receive button on the Transactions tab.
Macros
Examples
