Migrate environments to Cloud Composer 2 (from Airflow 1)

Console

Get the list of your Cloud Composer 1 environment's configuration overrides, custom PyPI packages, and environment variables:

1. Go to the Environments page in the Google Cloud console:

   Go to Environments

2. Select your Cloud Composer 1 environment.

3. View environment variables on the Environment variables tab.

4. View configuration overrides on the Airflow configurations overrides tabs.

5. View custom PyPI packages on the PyPI packages tab.

gcloud

To get the list of environment variables, run:

gcloud composer environments describe \
    COMPOSER_1_ENV \
    --location COMPOSER_1_LOCATION \
    --format="value(config.softwareConfig.envVariables)"

To get the list of environment's Airflow configuration overrides, run:

gcloud composer environments describe \
    COMPOSER_1_ENV \
    --location COMPOSER_1_LOCATION \
    --format="value(config.softwareConfig.airflowConfigOverrides)"

To get the list of custom PyPI packages, run:

gcloud composer environments describe \
    COMPOSER_1_ENV \
    --location COMPOSER_1_LOCATION \
    --format="value(config.softwareConfig.pypiPackages)"

Replace:

• COMPOSER_1_ENV with the name of your Cloud Composer 1 environment.
• COMPOSER_1_LOCATION with the region where the Cloud Composer 1 environment is located.

Terraform

Skip this step. Your Cloud Composer 1 environment's configuration already lists configuration overrides, custom PyPI packages, and environment variables for your environment.

Console

Create a Cloud Composer 2 environment and specify configuration overrides and environment variables.

As an alternative, you can override Airflow configurations and environment variables after you create an environment.

Some configuration options from Airflow 1 use a different name and section in Airflow 2. For more information, see Configuration changes.

gcloud

Create a Cloud Composer 2 environment and specify configuration overrides and environment variables.

As an alternative, you can override Airflow configurations and environment variables after you create an environment.

Some configuration options from Airflow 1 use a different name and section in Airflow 2. For more information, see Configuration changes.

Terraform

Create a Cloud Composer 2 environment based on the configuration of the Cloud Composer 1 environment:

1. Copy your Cloud Composer 1 environment's configuration.
2. Change the name of your environment.

3. Use the google-beta provider:

   resource "google_composer_environment" "example_environment_composer_2" {
     provider = google-beta
     # ...
   }
   

4. Specify a Cloud Composer 2 image in the config.software_config block:

   software_config {
     image_version = "composer-2.10.1-airflow-2.10.2"
     # ...
   }
   

5. If not aleady, specify configuration overrides and environment variables.

6. Specify custom PyPI packages in the config.software_config.pypi_packages block:

   software_config {
   
     # ...
   
     pypi_packages = {
       numpy = ""
       scipy = ">=1.1.0"
     }
   
   }
   

Console

1. Go to the Environments page in the Google Cloud console:

   Go to Environments

2. Select your Cloud Composer 2 environment.

3. Go to the PyPI packages tab and click Edit.

4. Copy PyPI package requirements from your Cloud Composer 1 environment. Click Save and wait until the environment updates.

gcloud

1. Create a requirements.txt file with the list of custom PyPI packages:

     numpy
     scipy>=1.1.0
   

2. Update your environment and pass the requirements.txt file in the :to the --update-pypi-packages-from-file command:

   gcloud composer environments update COMPOSER_2_ENV \
     --location COMPOSER_2_LOCATION  \
     --update-pypi-packages-from-file requirements.txt
   

   Replace:

   • COMPOSER_2_ENV with the name of your Cloud Composer 2 environment.
   • COMPOSER_2_LOCATION with the region where the Cloud Composer 2 environment is located.

Terraform

Skip this step. You already installed custom PyPI packages when you created the environment.

gcloud

1. Export variables from your Cloud Composer 1 environment:

   gcloud composer environments run \
       COMPOSER_1_ENV \
       --location COMPOSER_1_LOCATION \
        variables -- -e /home/airflow/gcs/data/variables.json
   

   Replace:

   • COMPOSER_1_ENV with the name of your Cloud Composer 1 environment.
   • COMPOSER_1_LOCATION with the region where the Cloud Composer 1 environment is located.

2. Export pools from your Cloud Composer 1 environment:

   gcloud composer environments run COMPOSER_1_ENV \
       --location COMPOSER_1_LOCATION \
        pool -- -e /home/airflow/gcs/data/pools.json
   

   Replace:

   • COMPOSER_1_ENV with the name of your Cloud Composer 1 environment.
   • COMPOSER_1_LOCATION with the region where the Cloud Composer 1 environment is located.

3. Get your Cloud Composer 2 environment's bucket URI.

   1. Run the following command:

      gcloud composer environments describe COMPOSER_2_ENV \
          --location COMPOSER_2_LOCATION \
           --format="value(config.dagGcsPrefix)"
      

      Replace:

      • COMPOSER_2_ENV with the name of your Cloud Composer 2 environment.
      • COMPOSER_2_LOCATION with the region where the environment is located.

   2. In the output, remove the /dags folder. The result is the URI of your Cloud Composer 2 environment's bucket.

      For example, change gs://us-central1-example-916807e1-bucket/dags to gs://us-central1-example-916807e1-bucket.

4. Transfer JSON files with variables and pools to your Cloud Composer 2 environment:

   gcloud composer environments storage data export \
       --destination=COMPOSER_2_BUCKET/data \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION \
       --source=variables.json
   

   gcloud composer environments storage data export \
       --destination=COMPOSER_2_BUCKET/data \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION \
       --source=pools.json
   

   Replace:

   • COMPOSER_2_BUCKET with the URI of your Cloud Composer 2 environment bucket, obtained on the previous step.
   • COMPOSER_1_ENV with the name of your Cloud Composer 1 environment.
   • COMPOSER_1_LOCATION with the region where the Cloud Composer 1 environment is located.

5. Import variables and pools to Cloud Composer 2:

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       variables import \
       -- /home/airflow/gcs/data/variables.json
   

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       pools import \
       -- /home/airflow/gcs/data/pools.json
   

6. Check that variables and pools are imported:

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       variables list
   

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       pools list
   

7. Remove JSON files from the buckets:

   gcloud composer environments storage data delete \
       variables.json \
       --environment=COMPOSER_2_ENV \
       --location=COMPOSER_2_LOCATION
   

   gcloud composer environments storage data delete \
       pools.json \
       --environment=COMPOSER_2_ENV \
       --location=COMPOSER_2_LOCATION
   

   gcloud composer environments storage data delete \
       variables.json \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION
   

   gcloud composer environments storage data delete \
       pools.json \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION
   

gcloud

1. Transfer plugins to your Cloud Composer 2 environment. To do so, export plugins from your Cloud Composer 1 environment's bucket to the /plugins folder in your Cloud Composer 2 environment's bucket:

   gcloud composer environments storage plugins export \
       --destination=COMPOSER_2_BUCKET/plugins \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION
   

2. Check that the /plugins folder is successfully imported:

   gcloud composer environments storage plugins list \
       --environment=COMPOSER_2_ENV \
       --location=COMPOSER_2_LOCATION
   

3. Export the /data folder from your Cloud Composer 1 environment to the Airflow 2 environment:

   gcloud composer environments storage data export \
       --destination=COMPOSER_2_BUCKET/data \
       --environment=COMPOSER_1_ENV \
       --location=COMPOSER_1_LOCATION
   

4. Check that the /data folder is successfully imported:

   gcloud composer environments storage data list \
       --environment=COMPOSER_2_ENV \
       --location=COMPOSER_2_LOCATION
   

gcloud

1. To get a list of connections in your Cloud Composer 1 environment, run:

   gcloud composer environments run COMPOSER_1_ENV \
       --location COMPOSER_1_LOCATION \
        connections -- --list
   

2. To create a new connection in your Cloud Composer 2 environment, run the connections Airflow CLI command through gcloud. For example:

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       connections add \
       -- --conn-host postgres.example.com \
       --conn-port 5432 \
       --conn-type postgres \
       --conn-login example_user \
       --conn-password example_password \
       --conn-description "Example connection" \
       example_connection
   

Airflow UI

1. To view a list of users in your Cloud Composer 1 environment:

   1. Open the Airflow web interface for your Cloud Composer 1 environment.

   2. Go to Admin > Users.

2. To create a user in your Cloud Composer 2 environment:

   1. Open the Airflow web interface for your Cloud Composer 2 environment.

   2. Go to Security > List Users.

   3. Click Add a new record.

gcloud

1. It is not possible to view a list of users through gcloud in Airflow 1. Please use the Airflow UI.

2. To create a new user account in your Cloud Composer 2 environment, run the users create Airflow CLI command through gcloud. For example:

   gcloud composer environments run \
       COMPOSER_2_ENV \
       --location COMPOSER_2_LOCATION \
       users create \
       -- --username example_username \
       --firstname Example-Name \
       --lastname Example-Surname \
       --email example-user@example.com \
       --use-random-password \
       --role Op
   

   Replace:

   • COMPOSER_2_ENV with the name of your Cloud Composer 2 environment.
   • COMPOSER_2_LOCATION with the region where the Cloud Composer 2 environment is located.
   • All user configuration parameters with their values from your Cloud Composer 1 environment, including user's role.