Move data between buckets

This page describes how to move the contents of a bucket by copying objects from a source bucket to a destination bucket.

Overview

When you create a bucket, some of the properties you set are permanent and cannot be changed, including the bucket's name and its parent project.

If you want to use a new name for your bucket, you can effectively do so by creating a new bucket with the name you want to use, and then copying your objects from the old bucket to the new bucket. You can then delete the old bucket. Note the following:

  • When you delete a bucket, the bucket's name can be reused.

  • If your primary goal of moving objects from one bucket to another is to store your data in a different bucket location, we recommend directly relocating your bucket.

  • Moving data between locations incurs network usage costs. In addition, moving data might incur additional charges, depending on the storage classes of the source data and on the location of the destination bucket. For more information, see Cloud Storage pricing.

Required permissions

Console

In order to complete this guide using the Google Cloud console, you must have the proper IAM permissions. If the buckets you want to access exist in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for the Google Cloud console.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

Command line

In order to complete this guide using a command-line utility, you must have the proper IAM permissions. If the buckets you want to access exist in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for gcloud storage commands.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

REST APIs

JSON API

In order to complete this guide using the JSON API, you must have the proper IAM permissions. If the buckets you want to access exist in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for JSON methods.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

Move data from one bucket to another

Console

Use the Storage Transfer Service from within Google Cloud console:

  1. If you don't have a destination bucket yet, create the bucket.

  2. Open the Transfer page in the Google Cloud console.

    Open the Transfer page

  3. Click Create transfer job.

  4. Follow the step-by-step walkthrough, clicking Next step as you complete each step:

    • Choose a source: Use Google Cloud Storage bucket as your source type, and either enter the name of the wanted bucket directly, or click Browse to find and select the bucket you want.

    • Choose a destination: Either enter the name of the wanted bucket directly, or click Browse to find and select the bucket you want.

    • Choose settings: Select the option Delete files from source after they're transferred.

    • Scheduling options: You can ignore this section.

  5. After you complete the step-by-step walkthrough, click Create.

    This begins the process of copying objects from your old bucket into your new one. This process may take some time; however, after you click Create, you can navigate away from the Google Cloud console.

    To view the transfer's progress:

    Open the Transfer page in the Google Cloud console.

    Open the Transfer page

    To learn how to get detailed error information about failed Cloud Storage operations in the Google Cloud console, see Troubleshooting.

  6. Once the transfer completes, you don't need to do anything to delete the objects from your old bucket if you selected the Delete source objects after the transfer completes checkbox during setup. You may, however, want to also delete your old bucket, which you must do separately.

Command line

  1. If you don't have one yet, create a destination bucket.

  2. To recursively copy the contents of your source bucket to your destination bucket, use the gcloud storage cp command with the --recursive option:

    gcloud storage cp --recursive gs://SOURCE_BUCKET/* gs://DESTINATION_BUCKET

    Where:

    • SOURCE_BUCKET is the name of your original bucket. For example, old-bucket.

    • DESTINATION_BUCKET is the name of the bucket you are moving your data to. For example, my-bucket.

    If your bucket uses managed folders and a flat namespace, you must include the flag --include-managed-folders in the command for the managed folders to be copied.

  3. To recursively delete the contents from the source bucket, along with the source bucket itself, use the gcloud storage rm command with the --recursive option:

    gcloud storage rm --recursive gs://SOURCE_BUCKET

    Where SOURCE_BUCKET is the name of your original bucket. For example, old-bucket.

    Alternatively, to delete the contents from the source bucket without deleting the source bucket itself, use the gcloud storage rm command with the --all-versions flag and ** wildcard:

    gcloud storage rm --all-versions gs://SOURCE_BUCKET/**

    Where SOURCE_BUCKET is the name of your original bucket. For example, old-bucket.

REST APIs

JSON API

  1. If you don't have a destination bucket yet, create the bucket.

  2. Copy each object from your source bucket to the destination bucket.

  3. Delete each object in your source bucket.

  4. Delete your source bucket.

XML API

  1. If you don't have a destination bucket yet, create the bucket.

  2. Copy each object from your source bucket to the destination bucket.

  3. Delete each object in your source bucket.

  4. Delete your source bucket.

What's next