Use a custom environment's bucket

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

You can specify a custom Cloud Storage bucket as an environment's bucket when you create an environment. Your environment uses this bucket in the same way as the default environment's bucket, and the default environment's bucket is not created.

With a custom bucket, you can:

  • Use a bucket configuration that is compliant with your requirements.
  • Use a continuously existing bucket for periodic development or testing purposes.
  • Reuse an existing bucket from a previous environment and keep CI/CD and data integrations.
  • Delete and re-create an environment without the need to transfer data between buckets.
  • Pre-populate the custom bucket with required data, such as your DAG files, and then create an environment.

Before you begin

  • This feature is supported starting from Cloud Composer version 2.5.0. Earlier versions of Cloud Composer 2 do not support this feature.
  • The custom bucket must be located in the same region as the environment. Multi-region and dual-region buckets are not supported.
  • The custom bucket and the environment must be in the same project.
  • The custom bucket must have the Standard storage class.
  • The custom bucket must have no retention policies or retention policy locks applied to it.

  • It is not possible to use the same custom bucket with several environments at the same time. Cloud Composer generates an error if the specified bucket is already used by another environment. It is possible to delete an environment and then use the same bucket for another environment.

  • If you enabled saving logs to the environment's bucket, then if you delete an environment and then use the same bucket for another environment, the new environment does not display Airflow task logs from the previous environment in Airflow UI. The logs remain in the bucket, but the new environment does not associate them with any DAG runs.

  • Cloud Composer doesn't add any missing IAM permissions to the Cloud Storage bucket when it is used to create an environment. Make sure that the environment's service account has permissions from the Composer Worker role on the bucket.

Create an environment with a custom bucket

Console

To specify a custom Cloud Storage bucket when you create an environment:

  1. On the Create environment page, in the Advanced configuration section, click Show advanced configuration.
  2. Select Custom bucket.
  3. In the Bucket name field, specify or select a bucket.

gcloud

When you create an environment, the --storage-bucket argument specifies the environment's custom bucket.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --storage-bucket CUSTOM_STORAGE_BUCKET

Replace the following:

  • ENVIRONMENT_NAME: the name of your environment.
  • LOCATION: the region where the environment is located.
  • CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket. You can also specify the bucket URI (with the gs:// prefix).

Example:

gcloud composer environments create example-environment \
    --location us-cental1 \
    --storage-bucket us-central1-example-bucket

API

When you create an environment, in the Environment > StorageConfig resource, specify a custom bucket for your environment.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "storageConfig": {
    "bucket": "CUSTOM_STORAGE_BUCKET"
  }
}

Replace the following:

  • PROJECT_ID: the Project ID.
  • LOCATION: the region where the environment is located.
  • ENVIRONMENT_NAME: the name of your environment.
  • CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket (without the gs:// prefix).

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "storageConfig": {
    "bucket": "us-central1-example-bucket"
  }
}

Terraform

When you create an environment, the bucket field in the storage_config block specifies the environment's custom bucket.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  storage_config {

    bucket = CUSTOM_STORAGE_BUCKET

  }
}

Replace the following:

  • ENVIRONMENT_NAME: the name of your environment.
  • LOCATION: the region where the environment is located.
  • CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket. You can also specify the bucket URI (with the gs:// prefix).

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  storage_config {

    bucket = "gs://us-central1-example-bucket"

  }
}

What's next