リリースノート

When providing a URL list to transfer files, you can now host the list itself in an access-controlled Cloud Storage bucket. See Transfer from public URLs for full details.

Storage Transfer Service now supports transfers from Amazon S3 over a Google-managed private network. Transfer jobs that select this option pay no AWS egress fees; instead, a flat per-GiB rate is charged by Google Cloud. This allows you to transfer data at a potentially lower overall cost.

Learn more about egress options for S3 transfers, including the managed private network.

Cloud Logging for Storage Transfer Service now supports transfers involving POSIX file systems.

See Cloud Logging for Storage Transfer Service for more details.

Storage Transfer Service has added support for Shared Keys as an authentication method when transferring from Microsoft Azure Storage.

To use an Azure Shared Key, you must store the key value in Secret Manager. See Save your Microsoft credentials in Secret Manager for details.

Support for transfers from cloud and on-premises Hadoop Distributed File System (HDFS) sources is now generally available (GA).

HDFS support allows for use cases such as migrating from on-premises storage to Cloud Storage, archiving data to free up on-premises storage space, replicating data to Google Cloud for business continuity, or transferring data to Google Cloud for analysis and processing.

See Transfer from HDFS to Cloud Storage for details.

Storage Transfer Service has added preview support for transferring managed folders between Cloud Storage buckets. Permissions on managed folders are copied between buckets when using this option.

See Transfer Cloud Storage managed folders for details.

Storage Transfer Service supports transfers from cloud and on-premises Hadoop Distributed File System (HDFS) sources. This feature is in limited release; complete the form linked from the user guide at Transfer from HDFS to Cloud Storage to request access.

This new feature supports use cases such as migrating from on-premises storage to Cloud Storage, archiving data to free up on-premises storage space, replicating data to Google Cloud for business continuity, or transferring data to Google Cloud for analysis and processing.

You can now transfer data from Amazon S3 via your CloudFront domain. Learn more.

Event-driven transfers are now generally available (GA).

Event-driven transfers provide serverless, real-time replication from AWS S3 to Cloud Storage, and between Cloud Storage buckets. With this new capability, you can accelerate your event-driven analytics pipeline, enable automatic replication across Cloud Storage buckets, create a backup copy of data in a different region or project, or perform live migration.

Learn more about Event-driven transfers.

Storage Transfer Service now offers Preview support for providing a Secret Manager secret when creating transfer jobs from AWS S3 or Azure storage.

Secret Manager provides strong encryption, role-based access control, and audit logging to protect your secrets.

For details, see the Secret Manager section of the following documents:

• Configure access to a source: Amazon S3
• Configure access to a source: Microsoft Azure Storage

Cloud Monitoring for Storage Transfer Service is now Generally Available (GA). This integration allows you to monitor the number of objects and data being transferred and to compute transfer speeds. This GA launch also adds monitoring of errors and error codes.

See Monitor transfer jobs for details.

Storage Transfer Service now publishes the IP ranges from which it makes requests to your AWS or Azure storage resources when performing a transfer. This allows you to restrict your resources by IP, and still allow Storage Transfer Service access.

For details, see the IP restrictions section of the following documents:

• Configure access to a source: Amazon S3
• Configure access to a source: Microsoft Azure Storage

Storage Transfer Service can now optionally preserve UID, GID, and mode metadata for folders, and recreate empty folders, when transferring between file systems.

See Metadata preservation for details.

Transfers from S3-compatible storage to Cloud Storage are now generally available (GA). This feature builds on support for Multipart upload and List Object V2, which makes Cloud Storage suitable for running applications written for the S3 API.

With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage. For customers moving data from AWS S3 to Cloud Storage, this feature provides an option to control network routes to Google Cloud, resulting in considerably lower egress charges.

See Transfer from S3-compatible sources for details.

Support for Manifest in Storage Transfer Service is now generally available (GA). You can use Manifest to transfer a specific list of objects, object versions, and files from cloud and on-premises sources. Programmatic users can use the output of an upstream operation generating a list of files and objects as an input for Storage Transfer Service to act upon.

Storage Transfer Service now offers Preview support for tracking progress of a Transfer Job using Cloud Monitoring, allowing you to monitor the number of objects and amount of data copied by Storage Transfer Service in near real-time.

See Monitor transfer jobs for details.

Storage Transfer Service now offers GA Support for transferring data between file systems, including on-premises file systems and Filestore instances. This allows you to use the Transfer Service API, gcloud command line tool, or the Cloud console to migrate data from a self-managed file system to Filestore; accelerate data transfer from an on-premise file system to a cloud file system; or move data between on-premises systems.

You can also transfer specific files or objects using a manifest for file system to file system transfers.

Storage Transfer Service offers Preview support for event-driven transfers - serverless, real-time replication from AWS S3 to Cloud Storage, and between Cloud Storage buckets. With this new capability, you can accelerate your event-driven analytics pipeline, enable automatic replication across Cloud Storage buckets, create a backup copy of data in a different region or project, or perform live migration.

Learn more about Event-driven transfers.

Multipart upload for transfers originating from a file system is now generally available (GA) and enabled by default.

Multipart uploads can speed up transfers that include large files. You must grant additional permissions on the destination or intermediate bucket; see Permissions for file system transfers for details.

The Storage Transfer Service REST API now provides a transferJobs.delete method. Deleting a transfer job stops all current and future operations associated with the job, and removes the job from all transfer job listings. Learn more on the Manage transfers page.

Support for exporting data from Cloud Storage to a file system is now generally available (GA).

You can use this bidirectional data movement capability to move data in and out of Cloud Storage, on-premises clusters, and edge locations including Google Distributed Cloud. To get started, see Requirements for file system transfers.

Storage Transfer Service now offers Preview support for moving data from S3-compatible storage to Cloud Storage. This feature builds on recent Cloud Storage launches, namely support for Multipart upload and List Object V2, which makes Cloud Storage suitable for running applications written for the S3 API. With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage. For customers moving data from AWS S3 to Cloud Storage, this feature provides an option to control network routes to Google Cloud, resulting in considerably lower egress charges.

See Transfer from S3-compatible sources for details.

Storage Transfer Service now offers preview support for multipart uploads for transfers originating from a file system, if the destination or intermediate bucket uses the Standard storage class.

Multipart uploads can speed up transfers that include large files. You must grant additional permissions on the destination or intermediate bucket; see Permissions for file system transfers for details.

Detailed logging for objects copied between AWS S3, Azure Blob Storage, ADLS Gen 2, and Cloud Storage with Storage Transfer Service is now generally available (GA).

With detailed logs of individual objects available in Cloud Logging, you can verify what was transferred and perform additional data integrity checks. This launch simplifies monitoring, reporting, and troubleshooting. Read Cloud Logging for Storage Transfer Service for details.

Expanded overwrite options are new generally available (GA). The overwriteWhen field can be used to specify whether data that already exists in the destination should be overwritten always, never, or only when ETags and checksum values indicate that the file has changed.

Metadata preservation options are now generally available (GA). This includes the option of preserving POSIX attributes and symlinks when transferring to, from, and between POSIX filesystems; as well as object ACLs, CMEK, temporary holds, and object creation time when transferring between Cloud Storage buckets.

See Metadata preservation for details.

Storage Transfer Service now provides more options for when to overwrite files that already exist in the destination. The new overwriteWhen field provides three options, that apply to all transfers, including those to or from file systems.

• NEVER provides defense in depth for archival cases, where data is not intended to be overwritten. Users no longer need to rely on a retention policy to protect their data.
• DIFFERENT uses ETags and checksum values to only overwrite a file if the contents have changed.
• ALWAYS overwrites any existing files with the same name. Avoids LIST operations on the destination when transferring into Cloud Storage.

Storage Transfer Service now offers a predefined role to simplify permission assignment to transfer agents. The roles/storagetransfer.transferAgent role grants a minimum set of permissions required for the service to communicate with agents and eliminates the need to assign each permission individually.

The role should be granted to the user account or service account being used to authenticate the agents. See the On-premises agent account documentation for more details.

Creating and managing data transfers with the gcloud command-line tool is now generally available (GA).

You can use gcloud commands to perform agent installation, manage agent pool lifecycles, and orchestrate transfer jobs. This launch simplifies writing scripts to automate transfer workflow.

The GA launch adds support for transfers between file systems, metadata preservation, and manifests. It also introduces the gcloud transfer authorize command to inspect and grant required permissions for transfers.

Storage Transfer Service now enforces the Resource Location Restriction, which is part of the Org Policy Service. This allows an organization to define the allowed regions in which location-based Google Cloud resources, such as transfer jobs, can be created.

To learn how Storage Transfer Service chooses the region in which to run a transfer job, refer to Location of Storage Transfer Service jobs.

Storage Transfer Service now offers more control over preserving metadata when transferring between Cloud Storage buckets. Choose to retain or discard metadata including object ACLs, customer-managed encryption keys (CMEK), temporary holds, and object creation time. In addition, storage class can be set to any supported value, allowing you to change storage class at scale.

For details, refer to Metadata preservation.

Storage Transfer Service now supports Cloud Client Libraries, which are the recommended option for accessing Cloud APIs programmatically. This launch significantly reduces the amount of code you need to write; see Getting started and Migrating to the Cloud Client Library for more details.

Support for agent pools is now generally available (GA) .

You can use agent pools to create isolated groups of agents as a source or sink entity in a transfer job. This enables you to transfer data from multiple data centers and filesystems concurrently, without creating multiple projects for a large transfer spanning multiple filesystems and data centers.

Storage Transfer Service now offers Preview support for moving data between two filesystems and keeping them in sync on a periodic schedule. Transfers can be orchestrated by creating a transfer job through the API or gcloud command-line tool. See Transfer data between POSIX file systems for details.

This launch offers a managed way to migrate from a self-managed filesystem to Filestore. If you have on-premises systems generating massive amounts of data that needs to be processed in Google Cloud, you can now use Storage Transfer Service to accelerate data transfer from an on-prem filesystem to a cloud filesystem.

Storage Transfer Service now offers Preview support for preserving POSIX attributes and symlinks when transferring to, from, and between POSIX filesystems. Attributes include the user ID of the owner, the group ID of the owning group, the mode or permissions, the modification time, and the size of the file. See Metadata preservation for details.

Integration with AWS Security Token Service is now generally available (GA) for Storage Transfer Service.

Security conscious customers can use Storage Transfer Service to perform transfers from AWS S3 without passing long-term AWS S3 credentials, which have to be rotated or explicitly revoked when they are no longer needed. Refer to the Amazon S3 > Federated Identity tab when setting up access to your data source.

Creating and managing data transfers with the gcloud command-line tool is now available in Preview. You can use gcloud commands to perform agent installation, manage agent pool lifecycles, and orchestrate transfer jobs. This launch simplifies writing scripts to automate transfer workflow.

Storage Transfer Service now offers Preview support for detailed logging for objects copied between AWS S3, Azure Blob, ADLS Gen 2, and Cloud Storage. With detailed logs of individual objects available in Cloud Logging, you can verify what was transferred and perform additional data integrity checks. This launch simplifies monitoring, reporting, and troubleshooting. Read Configure transfer logs for details.

Support for transferring data from Azure ADLS Gen 2 to Cloud Storage with Storage Transfer Service is now generally available (GA).

The Storage Transfer Service API for managing on-premises transfers is now generally available (GA). Customers can use RESTful APIs to automate their on-prem to Cloud transfer workflow.

For more information, see Managing Transfer for on-premises jobs.

Storage Transfer Service now offers preview support for Manifest. You can use Manifest to transfer a specific list of objects, object versions, and files from cloud and on-premises sources. Programmatic users can use the output of an upstream operation generating a list of files and objects as an input for Storage Transfer Service to act upon.

Storage Transfer Service now offers Preview support for agent pools. You can use agent pools to create isolated groups of agents as a source or sink entity in a transfer job. This enables you to transfer data from multiple data centers and filesystems concurrently, without creating multiple projects for a large transfer spanning multiple filesystems and data centers.

Storage Transfer Service now offers Preview support for exporting data from Cloud Storage to a POSIX file system. You can use this bidirectional data movement capability to move data in and out of Cloud Storage, on-premises clusters, and edge locations including Google Distributed Cloud. For more information, see Download data from Cloud Storage.

Storage Transfer Service now offers Public Preview support for managing on-premises transfer via API. Customers can use RESTful APIs to automate their on-prem to Cloud transfer workflow.

For more information, see Managing Transfer for on-premises jobs.

Transfer service for on-premises data support for delete from source is now Generally Available. For more information, see Data consistency details.

Storage Transfer Service offers Preview for Integration with AWS Security Token Service. Security conscious customers can now use Storage Transfer Service to perform transfers from AWS S3 without passing any security credentials. This release will alleviate the security burden associated with passing long-term AWS S3 credentials, which have to be rotated or explicitly revoked when they are no longer needed. Refer Amazon Web Services (AWS) S3 Federated Identity credentials when setting up access to your data source.

Storage Transfer Service offers Preview support for transferring data from Azure ADLS Gen 2 to Cloud Storage.

Obtaining the status of the latest transfer operation is in now generally available.

Storage Transfer Service support for hourly schedules and modifying a job's schedule is now generally available. For more information, see Schedule.

Storage Transfer Service support for specifying source and destination paths when creating a transfer is now generally available. For more information, see Specifying source and destination paths.

Storage Transfer Service now supports event-driven transfer workflows, a new capability that lets customers start Transfer jobs at once based on an event trigger. Customers can create Transfer jobs without specifying a schedule and start a TransferOperation using transferJobs.run from an existing TransferJob configuration.

Storage Transfer Service offers Preview support for specifying source and destination paths when creating a transfer. For more information, see Specifying source and destination paths.

Storage Transfer Service offers Preview support for hourly schedules and modifying a job's schedule. For more information, see Schedule.

Obtaining the status of the latest transfer operation is in Preview.

Storage Transfer Service integration for VPC Service Controls is now generally available.

Transfers from Microsoft Azure Blob Storage are now generally available.

Beta stage integration with VPC Service Controls

Transfer service on-premises: Beta stage integration with VPC Service Controls

Transfer Service for on-premises data is now generally available. For more information, see Transfer Service for on-premises data Overview.

Storage Transfer Service now offers Beta support for transfers from Microsoft Azure Blob Storage.

Storage Transfer Service supports Pub/Sub notifications for transfer jobs for external cloud provider to Cloud Storage transfers.

Storage Transfer Service now allows you to specify files to transfer based on their last modification times.

Storage Transfer Service supports custom job IDs.

Storage Transfer Service for on-premises is now in beta.

API documentation update to clarify transferJobs API descriptions, specifically: scheduleStartDate, scheduleEndDate, and startTimeOfDay.

Updates to API documentation.

Adding new IAM roles for Storage Transfer Service.

Updated documentation for configuring access to the data sink.

Documentation updates for IAM permissions.

Documentation update for access control.

Updates to API documentation.

Documentation update that provides clearer instructions for configuring a Cloud Storage data source.

Documentation update to add information for configuring access to a data sink and supported AWS regions.

API documentation update.

Documentation updates and content reorganization.

Updates to API documentation for scheduling jobs.

Documentation update to support scheduling recurring transfers with Cloud Console.

Updated Getting Started document for changes to the Create Transfer workflow.

Several updates to API documentation.

Documentation includes instructions for enabling API for non-UI users.

API documentation now includes "Try It" sections.

Documentation update for using prefixes in Storage Transfer Service user interface.

Storage Transfer Service, a cloud-to-cloud transfer tool, launched.