Cloud Composer release notes

Improved the error message generated when a Cloud Composer 3 environment creation fails because of missing permissions.

(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.25.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.24.0 to version 10.25.0.

(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.0 to version 9.0.1.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.10.2-build.0
• composer-3-airflow-2.9.3-build.7 (default)
• composer-3-airflow-2.7.3-build.23

New images are available in Cloud Composer 2:

• composer-2.9.11-airflow-2.10.2
• composer-2.9.11-airflow-2.9.3 (default)
• composer-2.9.11-airflow-2.7.3

Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.3 are supported until November 13, 2025.

(Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.6 (default)
• composer-3-airflow-2.7.3-build.22

New images are available in Cloud Composer 2:

• composer-2.9.10-airflow-2.9.3 (default)
• composer-2.9.10-airflow-2.7.3

Cloud Composer 2.9.7 is a version with an extended upgrade timeline.

(Airflow 2.9.3 and 2.7.3) The dbt-common package was downgraded from 1.11.0 to 1.10.0.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.5 (default)
• composer-3-airflow-2.7.3-build.21

Cloud Composer 2.9.9 images are available:

• composer-2.9.9-airflow-2.9.3 (default)
• composer-2.9.9-airflow-2.7.3

(Only new Cloud Composer 2 environments, all versions) If a GKE Control Plane IP range is specified for an environment, GKE creates a new subnetwork in this range to provision the IP address for communication with the GKE Control Plane. Otherwise, the subnetwork specified in the Cloud Composer connection subnetwork range is used (this range defaults to the environment's subnetwork). For more information about IP ranges used by Cloud Composer environments, see Configure private IP networking.

(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.24.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.23.0 to version 10.24.0.

(Airflow 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.2 to version 9.0.0.

(Airflow 2.9.3 and 2.7.3) Changes in preinstalled packages:

• The grpcio package was downgraded from 1.66.2 to 1.65.5.
• The js2py package was removed from dependencies.
• The pyjsparser package was removed from dependencies.
• The tzlocal package was removed from dependencies.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.4 (default)
• composer-3-airflow-2.7.3-build.20

Cloud Composer 2.9.8 images are available:

• composer-2.9.8-airflow-2.9.3 (default)
• composer-2.9.8-airflow-2.7.3

The default version of Airflow is changed to 2.9.3.

Airflow 2.9.1 is no longer included in Cloud Composer images and builds.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.3 (default)
• composer-3-airflow-2.7.3-build.19

Cloud Composer 2.9.7 images are available:

• composer-2.9.7-airflow-2.9.3 (default)
• composer-2.9.7-airflow-2.7.3

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.2
• composer-3-airflow-2.9.1-build.9 (default)
• composer-3-airflow-2.7.3-build.18

Cloud Composer 2.9.6 images are available:

• composer-2.9.6-airflow-2.9.3
• composer-2.9.6-airflow-2.9.1 (default)
• composer-2.9.6-airflow-2.7.3

(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-google package was upgraded to version 10.23.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.22.0 to version 10.23.0.

(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.2 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.1 to version 8.4.2.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.1
• composer-3-airflow-2.9.1-build.8 (default)
• composer-3-airflow-2.7.3-build.17

Cloud Composer 2.9.5 images are available:

• composer-2.9.5-airflow-2.9.3
• composer-2.9.5-airflow-2.9.1 (default)
• composer-2.9.5-airflow-2.7.3

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.3-build.0
• composer-3-airflow-2.9.1-build.7 (default)
• composer-3-airflow-2.7.3-build.16

Cloud Composer 2.9.4 images are available:

• composer-2.9.4-airflow-2.9.3
• composer-2.9.4-airflow-2.9.1 (default)
• composer-2.9.4-airflow-2.7.3

Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.1 are supported until September 18, 2025.

The apache-airflow-providers-google package was upgraded to version 10.22.0 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.22.0.

The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.1 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.3.4 to version 8.4.1.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.6 (default)
• composer-3-airflow-2.7.3-build.15

Cloud Composer 2.9.3 images are available:

• composer-2.9.3-airflow-2.9.1 (default)
• composer-2.9.3-airflow-2.7.3

The apache-airflow-providers-google package was upgraded to version 10.21.1 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.21.1.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.5
• composer-3-airflow-2.7.3-build.14

Cloud Composer 2.9.2 images are available:

• composer-2.9.2-airflow-2.9.1 (default)
• composer-2.9.2-airflow-2.7.3

(Airflow 2.7.3) Improved the reliability of connections between Airflow components and the Redis component.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.4
• composer-3-airflow-2.7.3-build.13

Cloud Composer 2.9.1 images are available:

• composer-2.9.1-airflow-2.9.1 (default)
• composer-2.9.1-airflow-2.7.3

(Airflow 2.9.1) Improved the reliability of connections between Airflow components and the Redis component.

The default version of Airflow is changed to 2.9.1.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.3
• composer-3-airflow-2.7.3-build.12

Cloud Composer 2.9.0 images are available:

• composer-2.9.0-airflow-2.9.1 (default)
• composer-2.9.0-airflow-2.7.3

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.2
• composer-3-airflow-2.7.3-build.11

Cloud Composer 2.8.8 images are available:

• composer-2.8.8-airflow-2.9.1
• composer-2.8.8-airflow-2.7.3 (default)

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.1
• composer-3-airflow-2.7.3-build.10

Cloud Composer 2.8.7 images are available:

• composer-2.8.7-airflow-2.9.1
• composer-2.8.7-airflow-2.7.3 (default)
• composer-2.8.7-airflow-2.6.3

Information about excluded Cloud Storage objects in the environment's bucket is no longer logged. This change reduces the usage of the Storage API during the synchronization of DAG files, which improves the performance of Airflow components and results in fewer Airflow component restarts. The change will gradually become available in all Cloud Composer environments.

(Airflow 2.9.1) Task context logging is disabled, and it is not possible to enable it.

(Airflow 2.9.1) Raw HTML code in DAG docs and DAG parameter descriptions is disabled by default.

(Airflow 2.9.1) Audit log permissions are revoked from all roles except Admin.

The apache-airflow-providers-google package was upgraded to version 10.21.0 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.18.0 to version 10.21.0.

The upgraded apache-airflow-providers-google package provides new operators and sensors for triggering DAGs across Cloud Composer environments:

• CloudComposerRunAirflowCLICommandOperator makes a call to the Airflow REST API of a Cloud Composer environment.
• CloudComposerDAGRunSensor checks if a DAG Run is completed.

When installing PyPI packages, if you want your builds to run with a custom service account, you can override the COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable with the chosen service account. For more information, see Install Python dependencies.

New Airflow builds are available in Cloud Composer 3:

• composer-3-airflow-2.9.1-build.0
• composer-3-airflow-2.7.3-build.9

Cloud Composer 2.8.6 images are available:

• composer-2.8.6-airflow-2.9.1
• composer-2.8.6-airflow-2.7.3 (default)
• composer-2.8.6-airflow-2.6.3

Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.7.3 are supported until July 17, 2025.

A new Airflow build is available in Cloud Composer 3:

• composer-3-airflow-2.7.3-build.8

Cloud Composer 2.8.5 images are available:

• composer-2.8.5-airflow-2.7.3 (default)
• composer-2.8.5-airflow-2.6.3

New Cloud Composer 2 environments are gradually switched to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. The IP address for the PSC endpoint will be taken from the nodes IP range. This change might require using a larger IP range for the nodes when you create an environment.

A new Airflow build is available in Cloud Composer 3:

• composer-3-airflow-2.7.3-build.7

Cloud Composer 2.8.4 images are available:

• composer-2.8.4-airflow-2.7.3 (default)
• composer-2.8.4-airflow-2.6.3

(Airflow 2.7.3) The apache-airflow-providers-google package was upgraded to version 10.18.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.17.0 to version 10.18.0.

(Airflow 2.7.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.3.1.

(Airflow 2.7.3) The apache-beam package was upgraded to version 2.56.0.

A new Airflow build is available in Cloud Composer 3:

• composer-3-airflow-2.7.3-build.6

Cloud Composer 2.8.3 images are available:

• composer-2.8.3-airflow-2.7.3 (default)
• composer-2.8.3-airflow-2.6.3

Cloud Composer 2.8.3 is a version with an extended upgrade timeline.

The google-cloud-bigquery package version was downgraded from 3.23.1 to 3.20.1 because of the #39541 issue in the public version of Airflow.

The dbt-core and dbt-bigquery packages were upgraded to version 1.8.1.

Cloud Composer 2.8.2 images are available:

• composer-2.8.2-airflow-2.7.3 (default)
• composer-2.8.2-airflow-2.6.3

Cloud Composer 2.8.1 images are available:

• composer-2.8.1-airflow-2.7.3 (default)
• composer-2.8.1-airflow-2.6.3

The Logs in Cloud Logging Only feature is enabled by default in new environments:

• New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
• Existing environments are not changed. If you upgrade an existing environment to Cloud Composer 2.8.0, it keeps saving logs to the environment's bucket.
• You can enable and disable saving logs to the environment's bucket for an existing environment.

Cloud Composer 2.8.0 images are available:

• composer-2.8.0-airflow-2.7.3 (default)
• composer-2.8.0-airflow-2.6.3

The apache-airflow-providers-google package is upgraded to version 10.17.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.16.0 to version 10.17.0.

The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.1.0.

Cloud Composer 2.7.1 images are available:

• composer-2.7.1-airflow-2.7.3 (default)
• composer-2.7.1-airflow-2.6.3

(New environments only) Increased the default value of the [webserver]auto_refresh_interval Airflow configuration option to 15 seconds. Pages in the Airflow UI, such as the list of DAGs, now will update every 15 seconds.

The default version of Airflow is changed to 2.7.3.

Cloud Composer 2.7.0 images are available:

• composer-2.7.0-airflow-2.7.3 (default)
• composer-2.7.0-airflow-2.6.3

The default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments. This change applies to new environments with Airflow 2.6.3 and 2.7.3 and to existing environments that do not use custom values for these parameters. For existing environments, the new default values will be used after an upgrade.

• [scheduler]job_heartbeat_sec to 30
• [scheduler]scheduler_health_check_threshold to 60
• [scheduler]scheduler_heartbeat_sec to 15

If you want to override the [scheduler]scheduler_heartbeat_sec option's value, then also adjust the [scheduler]scheduler_health_check_threshold option, as described in Cloud Composer documentation.

Cloud Composer 2.6.6 images are available:

• composer-2.6.6-airflow-2.7.3
• composer-2.6.6-airflow-2.6.3 (default)
• composer-2.6.6-airflow-2.5.3

The apache-airflow-providers-google package is upgraded to version 10.16.0 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version. For more information about changes, see the apache-airflow-providers-google changelog from version 10.15.0 to version 10.16.0.

The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.0.1 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version.

Cloud Composer 2.6.5 images are available:

• composer-2.6.5-airflow-2.7.3
• composer-2.6.5-airflow-2.6.3 (default)
• composer-2.6.5-airflow-2.5.3

Cloud Composer 2.6.5 is a version with an extended upgrade timeline.

Cloud Composer 2.6.4 images are available:

• composer-2.6.4-airflow-2.6.3 (default)
• composer-2.6.4-airflow-2.5.3

(Cloud Composer 2 in select regions) Cloud Composer shows the account selection and consent screens when opening Airflow UI for an environment.

If you have the "Don't allow users to access any third-party apps" setting enabled in Google Workspace, then configure access to the "Apache Airflow in Cloud Composer" app in Google Workspace, so that your Google Workspace users can still access Airflow UI in Cloud Composer environments. For more information, see Allow access to Airflow UI in Google Workspace.

(Cloud Composer 2 in select regions) Reduced the propagation time of the revoked Cloud IAM permission that blocks access to Airflow UI.

In new environments with Airflow 2.6.3, the default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments:

• [scheduler]job_heartbeat_sec to 30
• [scheduler]scheduler_heartbeat_sec to 15

The apache-airflow-providers-google package is upgraded to version 10.15.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.14.0 to version 10.15.0.

Cloud Composer 2.6.3 images are available:

• composer-2.6.3-airflow-2.6.3 (default)
• composer-2.6.3-airflow-2.5.3

Starting February 27, 2024, in the us-central1, europe-west1, europe-west2, europe-west3, europe-west6, us-east1, and us-east4 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.

In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.

Cloud Composer 2.6.2 images are available:

• composer-2.6.2-airflow-2.6.3 (default)
• composer-2.6.2-airflow-2.5.3

The apache-airflow-providers-google package is upgraded to version 10.14.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.13.1 to version 10.14.0.

Cloud Composer 2.6.1 images are available:

• composer-2.6.1-airflow-2.6.3 (default)
• composer-2.6.1-airflow-2.5.3

Data lineage integration is now enabled by default in newly created environments with Cloud Composer version 2.1.2 and later, if Data Lineage API is enabled in the environment's project. Existing and upgraded environments keep their current configuration.

(Available without upgrading) The default worker_concurrency in Airflow 2.6.3 and later versions is now calculated using a different formula. For more information, see Change worker concurrency.

Connections to the Redis environment component are now additionally secured with a password. Improved the reliability of the environment component responsible for metrics reporting (airflow-monitoring).

The apache-airflow-providers-google package is upgraded to version 10.13.1 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.12.0 to version 10.13.1.

Cloud Composer 2.6.0 images are available:

• composer-2.6.0-airflow-2.6.3 (default)
• composer-2.6.0-airflow-2.5.3

Cloud Composer 2.6.0 is a version with an extended upgrade timeline.

Sensitive Airflow configuration options are now hidden in Airflow UI by default. The default value of the [webserver]expose_config option is now set to non-sensitive-only.

Cloud Composer 2.5.4 images are available:

• composer-2.5.4-airflow-2.5.3
• composer-2.5.4-airflow-2.6.3 (default)

The apache-airflow-providers-google package is upgraded to version 10.12.0 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.11.1 to version 10.12.0.

Cloud Composer 2.5.3 images are available:

• composer-2.5.3-airflow-2.5.3
• composer-2.5.3-airflow-2.6.3 (default)

The apache-airflow-providers-google package is upgraded to version 10.11.1 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.10.1 to version 10.11.1.

The apache-airflow-providers-cncf-kubernetes package was upgraded to version 7.9.0.

Cloud Composer 2.5.2 images are available:

• composer-2.5.2-airflow-2.5.3
• composer-2.5.2-airflow-2.6.3 (default)

Quotas for snapshot operations are increased. Now it's possible to create up to 52 snapshots daily for a single environment. For scheduled snapshots taken at regular intervals, the minimum recommended interval is now 30 minutes (previously it was two hours).

The apache-airflow-providers-google package is upgraded to version 10.10.1 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.10.0 to version 10.10.1.

The apache-airflow-providers-cncf-kubernetes package was upgraded to version 7.6.0.

Cloud Composer 2.5.1 images are available:

• composer-2.5.1-airflow-2.5.3
• composer-2.5.1-airflow-2.6.3 (default)

Directories with names ending in .py are no longer synchronized. If your environment's bucket contains such directories, please rename them.

The apache-airflow-providers-google package is upgraded to version 10.10.0 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.9.0 to version 10.10.0.

The default version of Airflow is changed to 2.6.3.

Cloud Composer 2.5.0 images are available:

• composer-2.5.0-airflow-2.5.3
• composer-2.5.0-airflow-2.6.3 (default)

Airflow 2.6.3 consolidates the logic for handling tasks that are stuck in the queued state:

• The [kubernetes]worker_pods_pending_timeout, [celery]stalled_task_timeout, and [celery]task_adoption_timeout Airflow configuration options are deprecated and merged into the [scheduler]task_queued_timeout option.
• In Cloud Composer, the default value of the [scheduler]task_queued_timeout option is set to 40 minutes.
• If your environment uses a custom value for any of the deprecated Airflow configuration options, please clear the overrides before upgrading. If the values are not cleared, the longest timeout of all deprecated options is selected upon upgrading.
• If required, you can override the value of the [scheduler]task_queued_timeout option in your environment.
• For more information about other changes between Airflow versions 2.5.3 and 2.6.3, see Airflow release notes.

Cloud Composer 2.4.6 images are available:

• composer-2.4.6-airflow-2.6.3
• composer-2.4.6-airflow-2.5.3 (default)
• composer-2.4.6-airflow-2.4.3

Cloud Composer 2.4.5 images are available:

• composer-2.4.5-airflow-2.5.3 (default)
• composer-2.4.5-airflow-2.4.3

The cost of the environments.ExecuteAirflowCommand and environments.StopAirflowCommand operations is reduced from 100 to 25 quota units.

The apache-airflow-providers-google package is upgraded to version 10.9.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.7.0 to version 10.9.0.

Cloud Composer 2.4.4 images are available:

• composer-2.4.4-airflow-2.5.3 (default)
• composer-2.4.4-airflow-2.4.3

The apache-airflow-providers-google package is upgraded to version 10.7.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.5.0 to version 10.7.0.

Version 10.7.0 includes new Cloud Run, Cloud Batch and Dataplex Data Quality operators:

• Cloud Run is used to build and deploy scalable containerized apps written in any language (including Go, Python, Java, Node.js, .NET, and Ruby) on a fully managed platform. The newly provided Cloud Run Operators enable creating, executing, updating, deleting and listing Cloud Run Jobs.
• Cloud Batch is a fully managed batch service to schedule, queue, and execute batch jobs on Google's infrastructure. The provided Cloud Batch Operators enable submitting, listing and deleting batch jobs as well as listing a job's tasks.
• Dataplex is an intelligent data fabric that provides unified analytics and data management across your data lakes, data warehouses, and data marts. New Dataplex operators handle creating, updating, getting, deleting and running a Data Quality scan, getting a Data Quality Scan job, creating and deleting a zone, as well as creating and deleting an asset.

Cloud Composer 2.4.3 images are available:

• composer-2.4.3-airflow-2.5.3 (default)
• composer-2.4.3-airflow-2.4.3

Cloud Composer now generates a FAILED_PRECONDITION error message when an environment operation fails because the environment's cluster is missing.

Cloud Composer 2.4.2 images are available:

• composer-2.4.2-airflow-2.5.3 (default)
• composer-2.4.2-airflow-2.4.3

(Cloud Composer 2) An update that enforces VPC Service Controls perimeter settings for accessing Airflow UI and Airflow REST API is rolled out to all regions. As a result of this change, certain users that could previously access the Airflow UI or Airflow REST API might not be able to do it now.

If you experience problems with access to Airflow UI or Airflow REST API, update the perimeter or access level configuration for affected users. For more information on configuring VPC Service Controls see Configure VPC Service Controls.

The apache-airflow-providers-google package is upgraded to version 10.5.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.3.0 to version 10.5.0.

Cloud Composer 2.4.1 images are available:

• composer-2.4.1-airflow-2.5.3 (default)
• composer-2.4.1-airflow-2.4.3

The composer.environments.executeAirflowCommand permission is now required to run Airflow CLI commands through the gcloud environments run command:

• The composer.user and composer.environmentAndStorageObjectViewer roles do not have this permission and are not permitted to run Airflow CLI commands.

• This permission is added to the composer.admin and composer.environmentAndStorageObjectAdmin roles.

GCSFuse version updated to 1.01. Cloud Composer uses GCSFuse to sync files between the environment bucket and worker pods.

Cloud Composer 2.4.0 images are available:

• composer-2.4.0-airflow-2.5.3 (default)
• composer-2.4.0-airflow-2.4.3

The apache-airflow-providers-google package is upgraded to public version 10.3.0 in images with Airflow 2.5.3 and 2.4.3. Changes compared to version 2023.6.6+composer:

• Google Ads API default version updated to v14.
• scipy package is removed from dependencies.
• Several other small fixes and improvements.

For more information about changes, see the apache-airflow-providers-google changelog from version 10.1.1 to version 10.3.0.

(Cloud Composer 2) The default version of Airflow is changed to 2.5.3.

Cloud Composer 2.3.5 images are available:

• composer-2.3.5-airflow-2.5.3 (default)
• composer-2.3.5-airflow-2.4.3

This version is a version with an extended upgrade timeline.

(Available without upgrading) The default amount of memory available to the web server, schedulers, and workers is changed from 3.75 GB per CPU core to 4 GB per CPU core. This value is used only if you do not specify the amount of memory available to these environment components.

(Available without upgrading) Improved the performance of DAG UI in Private IP environments.

Cloud Composer 2.3.4 images are available:

• composer-2.3.4-airflow-2.5.3
• composer-2.3.4-airflow-2.5.1 (default)
• composer-2.3.4-airflow-2.4.3

Improved the error message when environment creation fails because of issues with GKE workloads.

Cloud Composer 2.3.3 images are available:

• composer-2.3.3-airflow-2.5.1 (default)
• composer-2.3.3-airflow-2.4.3

(Airflow 2.5.1) Changed the severity of triggerer watchdog messages from error to warning and updated the message's content to be more informative. This change improves debugging experience for triggers.

Cloud Composer 2.3.2 images are available:

• composer-2.3.2-airflow-2.5.1 (default)
• composer-2.3.2-airflow-2.4.3

Cloud Composer 2.3.1 images are available:

• composer-2.3.1-airflow-2.5.1 (default)
• composer-2.3.1-airflow-2.4.3

The apache-airflow-providers-google package in images with Airflow 2.5.1 and 2.4.3 was upgraded to version 2023.6.6+composer. This version is based on the public version 10.1.1, with additional fixes to some operators and upgrades to many SDK package dependencies (such as protobuf). Notable changes include:

• Google Ads default API changed from version 12 to 13.
• protobuf==4.22.5 is included, this is the first Cloud Composer version with protobuf version 4.x.
• In this version of the provider package, the deprecated delegate_to parameter is removed from all GCP operators, hooks, and triggers, as well as from Firestore and Gsuite transfer operators that interact with GCS. Impersonation can be achieved by utilizing the impersonation_chain parameter instead. The delegate_to param remains available only in Gsuite and marketing platform hooks and operators that don't interact with Google Cloud.

For a full list of changes in the apache-airflow-providers-google, see the changelog from version 8.9.0 to 10.1.1 on the apache-airflow-providers-google page.

The google-cloud-asset package is added to images with Airflow 2.5.1 and 2.4.3.

Cloud Composer 2.3.0 images are available:

• composer-2.3.0-airflow-2.5.1 (default)
• composer-2.3.0-airflow-2.4.3

(Cloud Composer 2) The deprecated [core]non_pooled_task_slot_count Airflow configuration option is replaced with the [core]default_pool_task_slot_count configuration option in the default Airflow configuration. Make sure to update your custom Airflow configuration overrides to use the new option instead of the deprecated one.

An improved error message is now displayed when a subnetwork with unsupported IPv4 ranges is used to create an environment in a shared VPC configuration.

Cloud Composer 2.2.1 images are available:

• composer-2.2.1-airflow-2.5.1 (default)
• composer-2.2.1-airflow-2.4.3

(Cloud Composer 2) Increased the default values of [core]dagbag_import_timeout to 120 seconds, and dag_file_processor_timeout to 300 seconds. The updated parameters allow the DAG Processor to parse more DAGs and provide more time to add parsed DAGs into the DAG bag.

(Cloud Composer 2) Increased the default value of [scheduler]zombie_detection_interval to 20 seconds. This change reduces the chance that Airflow skips retry attempts for a zombie task.

Cloud Composer 2.2.0 images are available:

• composer-2.2.0-airflow-2.5.1 (default)
• composer-2.2.0-airflow-2.4.3

(Cloud Composer 2) The default [celery]stalled_task_timeout value is set to 1200. Tasks that stay in the queued state for more than 20 minutes (1200 seconds) are now rescheduled.

(Airflow 2.4.3 and 2.5.1) Python packages upgraded:

• Added new packages: dbt-bigquery and google-cloud-documentai.
• The apache-airflow-providers-cncf-kubernetes package was upgraded to version 6.0.0.

(Cloud Composer 2) The default version of Airflow is changed to 2.5.1.

Cloud Composer 2.1.15 images are available:

• composer-2.1.15-airflow-2.5.1 (default)
• composer-2.1.15-airflow-2.4.3

Cloud Composer 2.1.14 and 1.20.12 images are available:

• composer-2.1.14-airflow-2.5.1
• composer-2.1.14-airflow-2.4.3 (default)
• composer-2.1.14-airflow-2.3.4
• composer-1.20.12-airflow-2.4.3
• composer-1.20.12-airflow-2.3.4
• composer-1.20.12-airflow-1.10.15

The apache-airflow-providers-google package in images with Airflow 2.3.4 and 2.4.3 was upgraded to 2023.4.13+composer. Changes compared to version 2023.3.14+composer:

• Update Google Display and Video 360 operators to use API v2.
• Update Google Campaign Manager operators to use API v4.
• Update google-cloud-dlp package to version 3.7.1 and adjust hooks and operators.

Cloud Composer 2.1.13 images are available:

• composer-2.1.13-airflow-2.4.3 (default)
• composer-2.1.13-airflow-2.3.4

GCSFuse version updated to 42.3. Cloud Composer uses GCSFuse to sync files between the environment bucket and worker pods.

Cloud Composer 2.1.12 images are available:

• composer-2.1.12-airflow-2.4.3 (default)
• composer-2.1.12-airflow-2.3.4

The default value for the dag_dir_list_interval Airflow configuration option is changed from 30 to 120 seconds.

Increased the timeout for environment operations performed by Cloud Build to 35 minutes.

Cloud Composer 2.1.11 and 1.20.11 images are available:

• composer-2.1.11-airflow-2.4.3 (default)
• composer-2.1.11-airflow-2.3.4
• composer-1.20.11-airflow-1.10.15
• composer-1.20.11-airflow-2.4.3
• composer-1.20.11-airflow-2.3.4

(Airflow 2.3.4 and 2.4.3 only) The Google Ads Python library (google-ads) package is updated to version 20.0.0.

(Airflow 2.3.4 and 2.4.3 only) The apache-airflow-providers-google package was upgraded to version 2023.3.14+composer. This version is based on the public version 8.9.0 with the following changes:

• Use google-ads==20.0.0

The source code for the apache-airflow-providers-google package version 2023.3.14+composer is available on GitHub.

(Composer 2 only) The instance_name field for Airflow Webservers in new environments is now by default set to the Composer Environment name.

Cloud Composer 2.1.10 and 1.20.10 images are available:

• composer-2.1.10-airflow-2.4.3 (default)
• composer-2.1.10-airflow-2.3.4
• composer-1.20.10-airflow-1.10.15
• composer-1.20.10-airflow-2.4.3
• composer-1.20.10-airflow-2.3.4

Cloud Composer 1.20.1 and 2.1.11 are versions with an extended upgrade timeline.

Cloud Composer 2.1.9 and 1.20.9 images are available:

• composer-2.1.9-airflow-2.4.3 (default)
• composer-2.1.9-airflow-2.3.4
• composer-2.1.9-airflow-2.2.5
• composer-1.20.9-airflow-1.10.15
• composer-1.20.9-airflow-2.4.3
• composer-1.20.9-airflow-2.3.4
• composer-1.20.9-airflow-2.2.5

Cloud Composer 2.1.8 and 1.20.8 images are available:

• composer-2.1.8-airflow-2.4.3 (default)
• composer-2.1.8-airflow-2.3.4
• composer-2.1.8-airflow-2.2.5
• composer-1.20.8-airflow-1.10.15
• composer-1.20.8-airflow-2.4.3
• composer-1.20.8-airflow-2.3.4
• composer-1.20.8-airflow-2.2.5

The default Composer version is changed to Cloud Composer 2.

The default Cloud Composer image used for operations is now the latest version of Cloud Composer 2, unless the image-version parameter is explicitly specified.

The composer-latest-airflow-* version aliases now point to Cloud Composer 2.

The apache-airflow-providers-google package is upgraded to the public version 8.9.0 in images with Airflow 2.4.3 and 2.3.4. For more information about changes, see the apache-airflow-providers-google page.

Cloud Composer 2.1.7 and 1.20.7 images are available:

• composer-2.1.7-airflow-2.4.3 (default)
• composer-2.1.7-airflow-2.3.4
• composer-2.1.7-airflow-2.2.5
• composer-1.20.7-airflow-1.10.15
• composer-1.20.7-airflow-2.4.3
• composer-1.20.7-airflow-2.3.4
• composer-1.20.7-airflow-2.2.5

(Cloud Composer 2) The default version of Airflow is changed to 2.4.3.

Cloud Composer 2.1.6 and 1.20.6 images are available:

• composer-2.1.6-airflow-2.4.3 (default)
• composer-2.1.6-airflow-2.3.4
• composer-2.1.6-airflow-2.2.5
• composer-1.20.6-airflow-1.10.15 (default)
• composer-1.20.6-airflow-2.4.3
• composer-1.20.6-airflow-2.3.4
• composer-1.20.6-airflow-2.2.5

Cloud Composer 1.20.5 and 2.1.5 images are available:

• composer-1.20.5-airflow-1.10.15 (default)
• composer-1.20.5-airflow-2.2.5
• composer-1.20.5-airflow-2.3.4
• composer-1.20.5-airflow-2.4.3
• composer-2.1.5-airflow-2.2.5
• composer-2.1.5-airflow-2.3.4 (default)
• composer-2.1.5-airflow-2.4.3

Images with Airflow 2.4.3 use the public version 8.6.0 of the apache-airflow-providers-google package. For more information about changes, see the package's page in the Airflow documentation.

(Airflow 2.2.5 only) The apache-airflow-providers-google package in images with Airflow 2.2.5 was upgraded to 2022.12.20+composer. Changes compared to version 2022.10.17+composer:

• Fix issue with the recursive error #25898 and #26175.
• Change log level to DEBUG for messages generated when a secret is not found in Secret Manager.
• Improve memory usage in Dataproc deferrable operators.

Cloud Composer 1.20.4 and 2.1.4 images are available:

• composer-1.20.4-airflow-1.10.15 (default)
• composer-1.20.4-airflow-2.2.5
• composer-1.20.4-airflow-2.3.4
• composer-1.20.4-airflow-2.4.3
• composer-2.1.4-airflow-2.2.5
• composer-2.1.4-airflow-2.3.4 (default)
• composer-2.1.4-airflow-2.4.3

Cloud Composer 1.20.3 and 2.1.3 images are available:

• composer-1.20.3-airflow-1.10.15 (default)
• composer-1.20.3-airflow-2.2.5
• composer-1.20.3-airflow-2.3.4
• composer-2.1.3-airflow-2.2.5
• composer-2.1.3-airflow-2.3.4 (default)

Cloud Composer 1.20.2 and 2.1.2 are versions with an extended upgrade timeline.

Cloud Composer 1.20.2 and 2.1.2 images are available:

• composer-1.20.2-airflow-1.10.15 (default)
• composer-1.20.2-airflow-2.2.5
• composer-1.20.2-airflow-2.3.4
• composer-2.1.2-airflow-2.2.5
• composer-2.1.2-airflow-2.3.4 (default)

(Available without upgrading) Allowed custom secondary IP range for pods is now narrower. You can now create Composer environments using IP ranges with /23 masks.

(New environments only) Airflow 2.3.4 is now the default Airflow version for Composer 2 environments.

(Cloud Composer 2) Starting with version 2.1.0, Composer environments no longer use Deployment Manager.

Images with Airflow 2.3.4 use the public version 8.4.0 of the apache-airflow-providers-google package. See the package page.

(Cloud Composer 2 with Private Service Connect) SQL proxy now gets restarted faster when connection issues are detected.

(Cloud Composer 2) You can now create Composer environments with labels that contain only digits.

Cloud Composer 1.19.15 and 2.0.32 images are available:

• composer-1.19.15-airflow-1.10.15 (default)
• composer-1.19.15-airflow-2.1.4
• composer-1.19.15-airflow-2.2.5
• composer-1.19.15-airflow-2.3.4
• composer-2.0.32-airflow-2.1.4
• composer-2.0.32-airflow-2.2.5
• composer-2.0.32-airflow-2.3.4

Cloud Composer 1.19.14 and 2.0.31 images are available:

• composer-1.19.14-airflow-1.10.15 (default)
• composer-1.19.14-airflow-2.1.4
• composer-1.19.14-airflow-2.2.5
• composer-1.19.14-airflow-2.3.3
• composer-2.0.31-airflow-2.1.4
• composer-2.0.31-airflow-2.2.5
• composer-2.0.31-airflow-2.3.3

The apache-airflow-providers-google package in images with Airflow 2.1.4 and 2.2.5 was upgraded to 2022.10.17+composer. Changes compared to version 2022.9.6+composer:

• Upgraded requirements for the google-cloud-container package from >=0.1.1,<2.0.0 to >=2.2.0,<3.0.0.
• Added changes for GKEHook and GKE unit tests from #22852, without pulling changes for DataprocCreateClusterOperator.

Cloud Composer 1.19.13 and 2.0.30 images are available:

• composer-1.19.13-airflow-1.10.15 (default)
• composer-1.19.13-airflow-2.1.4
• composer-1.19.13-airflow-2.2.5
• composer-1.19.13-airflow-2.3.3
• composer-2.0.30-airflow-2.1.4
• composer-2.0.30-airflow-2.2.5
• composer-2.0.30-airflow-2.3.3

Starting from Cloud Composer version 2.0.29, new private IP environments by default use Private Service Connect (instead of VPC peerings).

You can set HTTP_PROXY and HTTPS_PROXY environment variables in your environment. These standard Linux variables are used to route traffic through the specified proxies. The NO_PROXY variable by default is set to a list of Google domains so that they are excluded from proxying.

The result of the airflow_monitoring DAG is no longer automatically sent to XCom. This change improves monitoring performance.

Cloud Composer 1.19.12 and 2.0.29 images are available:

• composer-1.19.12-airflow-1.10.15 (default)
• composer-1.19.12-airflow-2.1.4
• composer-1.19.12-airflow-2.2.5
• composer-1.19.12-airflow-2.3.3
• composer-2.0.29-airflow-2.1.4
• composer-2.0.29-airflow-2.2.5
• composer-2.0.29-airflow-2.3.3

Images with Airflow 2.3.3 use the public version 8.1.0 of the apache-airflow-providers-google package. Other Cloud Composer images still use a custom version of this package, which is based on the public version 6.8.0.

When upgrading to Airflow 2.3, make sure that your environment is not affected by breaking changes introduced in the version 7.0.0 of this package.

Version 8.1.0 does not include some connectors, such as Dataform. Support for Dataform connectors will be available starting from version 8.3.0, which will be released together with Cloud Composer images with Airflow 2.3.4.

Cloud Composer 1.19.11 and 2.0.28 images are available:

• composer-1.19.11-airflow-1.10.15 (default)
• composer-1.19.11-airflow-2.1.4
• composer-1.19.11-airflow-2.2.5
• composer-1.19.11-airflow-2.3.3
• composer-2.0.28-airflow-2.1.4
• composer-2.0.28-airflow-2.2.5
• composer-2.0.28-airflow-2.3.3

The apache-airflow-providers-google package was upgraded to 2022.9.6+composer. Changes compared to version 2022.8.26+composer:

• Fix the Triggerer's async thread was blocked error in the deferrable mode for Composer Operators. (#25951)
• Fix a delay in Dataproc CreateBatch operator (#26126)
• Show better log for Datafusion operators when the response is empty (#26202)

Cloud Composer 1.19.10 and 2.0.27 images are available:

• composer-1.19.10-airflow-1.10.15 (default)
• composer-1.19.10-airflow-2.1.4
• composer-1.19.10-airflow-2.2.5
• composer-2.0.27-airflow-2.1.4
• composer-2.0.27-airflow-2.2.5

(Available without upgrading) When an environment is deleted, Cloud Composer automatically deletes the persistent disk of the environment's Redis queue.

(Available without upgrading) Cloud Composer now makes several attempts to create an environment when the Cloud Composer connection subnetwork is locked by another operation. This change improves the reliability when creating environments with Public Service Connect.

The apache-airflow-providers-google package was upgraded to 2022.8.26+composer. Changes compared to version 2022.8.23+composer:

• Added deferrable option to Dataproc operators to run the task asynchronously. (#25302)

  Cloud Composer team plans to add support for Deferrable operators in October, 2022.

Cloud Composer 1.19.9 and 2.0.26 images are available:

• composer-1.19.9-airflow-1.10.15 (default)
• composer-1.19.9-airflow-2.1.4
• composer-1.19.9-airflow-2.2.5
• composer-2.0.26-airflow-2.1.4
• composer-2.0.26-airflow-2.2.5

The apache-airflow-providers-google package was upgraded to 2022.8.23+composer. Changes compared to version 2022.8.16+composer:

• Added missing google-cloud-dataform package.

Cloud Composer uses a custom version of the apache-airflow-providers-google package. This custom version is based on the public version 6.8.0. For information about other changes compared to version 6.8.0, see release notes for the previous versions of this package.

(Available without upgrading) It is no longer possible to set environment variables with names http_proxy and https_proxy.

Cloud Composer 1.19.8 and 2.0.25 images are available:

• composer-1.19.8-airflow-1.10.15 (default)
• composer-1.19.8-airflow-2.1.4
• composer-1.19.8-airflow-2.2.5
• composer-2.0.25-airflow-2.1.4
• composer-2.0.25-airflow-2.2.5

The apache-airflow-providers-google package was upgraded to 2022.8.16+composer. Changes compared to version 2022.6.22+composer:

• Fix Vertex AI Custom Job training issue (#25367)
• New Operators for the Google Cloud Dataform service (#25587)

Cloud Composer uses a custom version of the apache-airflow-providers-google package. This custom version is based on the public version 6.8.0. For information about other changes compared to version 6.8.0, see release notes for the previous versions of this package.

The following packages are now preinstalled in Cloud Composer images with Airflow 2.2.5: google-cloud-firestore, firebase-admin, gcfs, dbt-core, apache-airflow-providers-dbt-cloud.

Cloud Composer 1.19.7 and 2.0.24 images are available:

• composer-1.19.7-airflow-1.10.15 (default)
• composer-1.19.7-airflow-2.1.4
• composer-1.19.7-airflow-2.2.5
• composer-2.0.24-airflow-2.1.4
• composer-2.0.24-airflow-2.2.5

Logs that are generated when processing individual DAG files are no longer written to the Airflow scheduler's disk. This change improves the scheduler's disk usage.

Cloud Composer 1.19.6 and 2.0.23 images are available:

• composer-1.19.6-airflow-1.10.15 (default)
• composer-1.19.6-airflow-2.1.4
• composer-1.19.6-airflow-2.2.5
• composer-2.0.23-airflow-2.1.4
• composer-2.0.23-airflow-2.2.5

Cloud Composer 1.19.5 and 2.0.22 images are available:

• composer-1.19.5-airflow-1.10.15 (default)
• composer-1.19.5-airflow-2.1.4
• composer-1.19.5-airflow-2.2.5
• composer-2.0.22-airflow-2.1.4
• composer-2.0.22-airflow-2.2.5

Cloud Composer 1.19.4 and 2.0.21 images are available:

• composer-1.19.4-airflow-1.10.15 (default)
• composer-1.19.4-airflow-2.1.4
• composer-1.19.4-airflow-2.2.5
• composer-2.0.21-airflow-2.1.4
• composer-2.0.21-airflow-2.2.5

Source code for the apache-airflow-providers-google package versions 2022.6.22+composer and 2022.5.18+composer is available on GitHub:

• 2022.6.22+composer
• 2022.5.18+composer

Cloud Composer 1.19.3 and 2.0.20 images are available:

• composer-1.19.3-airflow-1.10.15 (default)
• composer-1.19.3-airflow-2.1.4
• composer-1.19.3-airflow-2.2.5
• composer-2.0.20-airflow-2.1.4
• composer-2.0.20-airflow-2.2.5

Cloud Composer 1.19.2 and 2.0.19 are versions with an extended upgrade timeline.

Cloud Composer 1.19.2 and 2.0.19 images are available:

• composer-1.19.2-airflow-1.10.15 (default)
• composer-1.19.2-airflow-2.1.4
• composer-1.19.2-airflow-2.2.5
• composer-2.0.19-airflow-2.1.4
• composer-2.0.19-airflow-2.2.5

The apache-airflow-providers-google package upgraded to 2022.6.22+composer. Changes compared to version 2022.5.18+composer:

• Fixed a regression in BigQueryToGCPOpertor after changes to links were introduced in #24416.
• Fixed errors related to the usage of the service_account attribute by BeamRunJavaPipelineOperator.

Cloud Composer uses a custom version of the apache-airflow-providers-google package. This custom version is based on the public version 6.8.0. For information about other changes compared to version 6.8.0, see release notes for the previous versions of this package.

Cloud Composer 1.19.1 and 2.0.18 images are available:

• composer-1.19.1-airflow-1.10.15 (default)
• composer-1.19.1-airflow-2.1.4
• composer-1.19.1-airflow-2.2.5
• composer-2.0.18-airflow-2.1.4
• composer-2.0.18-airflow-2.2.5

Cloud Composer 1.18.12 is a version with an extended upgrade timeline.

Cloud Composer 1.19.0 and 2.0.17 images are available:

• composer-1.19.0-airflow-1.10.15 (default)
• composer-1.19.0-airflow-2.1.4
• composer-1.19.0-airflow-2.2.5
• composer-2.0.17-airflow-2.1.4
• composer-2.0.17-airflow-2.2.5

(Airflow 1.10.15) Upgraded apache-beam and google provider packages to version 2022.6.1:

• Support impersonation_chain parameter for Dataflow runner in Apache Beam operators
• Added missing project_id parameter for wait_for_job method in the Dataflow operators
• Added key_secret_project_id parameter which specifies a project with KeyFile

Cloud Composer 1.18.12 and 2.0.16 images are available:

• composer-1.18.12-airflow-1.10.15 (default)
• composer-1.18.12-airflow-2.1.4
• composer-1.18.12-airflow-2.2.5
• composer-2.0.16-airflow-2.1.4
• composer-2.0.16-airflow-2.2.5

Cloud Composer 1.18.11 and 2.0.15 images are available:

• composer-1.18.11-airflow-1.10.15 (default)
• composer-1.18.11-airflow-2.1.4
• composer-1.18.11-airflow-2.2.5
• composer-2.0.15-airflow-2.1.4
• composer-2.0.15-airflow-2.2.5

(Airflow 1) The google-cloud-bigquery package is upgraded from 1.28.0 to 2.13.0.

Cloud Composer uses a custom version of the apache-airflow-providers-google package, 2022.5.18+composer. This custom version is based on the public version 6.8.0.

Changes compared to version 6.8.0:

Breaking changes:

• Upgrade to support Google Ads v10 (#22965)

Features:

• [FEATURE] google provider - BigQueryInsertJobOperator log query (#23648)
• [FEATURE] google provider - split GkeStartPodOperator execute (#23518)
• Add exportContext.offload flag to CLOUD_SQL_EXPORT_VALIDATION. (#23614)
• Create links for BiqTable operators (#23164)
• implements #22859 - Add .sql as templatable extension (#22920)
• GCSFileTransformOperator: New templated fields 'source_object', 'destination_object' (#23328)

Bug Fixes

• Fix PostgresToGCSOperator does not allow nested JSON (#23063)
• Fix GCSToGCSOperator ignores replace parameter when there is no wildcard (#23340)
• update processor to fix broken download URLs (#23299)
• LookerStartPdtBuildOperator, LookerCheckPdtBuildSensor : fix empty materialization id handling (#23025)
• Change ComputeSSH to throw provider import error instead paramiko (#23035)
• Fix cancel_on_kill after execution timeout for DataprocSubmitJobOperator (#22955)
• Fix select * query xcom push for BigQueryGetDataOperator (#22936)
• MSSQLToGCSOperator fails: datetime is not JSON Serializable (#22882)
• Update credentials when using ADC in Compute Engine #23773

Misc changes

• Add Stackdriver assets and migrate system tests to AIP-47 (#23320)
• CloudTasks assets & system tests migration (AIP-47) (#23282)
• TextToSpeech assets & system tests migration (AIP-47) (#23247)
• Fix code-snippets in google provider (#23438)
• BigQuery assets (#23165)
• Remove redundant docstring in BigQueryUpdateTableSchemaOperator (#23349)
• Migrate gcs to new system tests design (#22778)
• add missing docstring in 'BigQueryHook.create_empty_table' (#23270)
• Cleanup Google provider CHANGELOG.rst (#23390)
• migrate system test gcs_to_bigquery into new design (#22753)
• Add example DAG for demonstrating usage of GCS sensors (#22808)

Cloud Composer 1.18.10 and 2.0.14 images are available:

• composer-1.18.10-airflow-1.10.15 (default)
• composer-1.18.10-airflow-2.1.4
• composer-1.18.10-airflow-2.2.3
• composer-1.18.10-airflow-2.2.5
• composer-2.0.14-airflow-2.1.4
• composer-2.0.14-airflow-2.2.3
• composer-2.0.14-airflow-2.2.5

If it is not possible to create an environment because of CMEK-related organization policies constraints/gcp.restrictCmekCryptoKeyProjects and constraints/gcp.restrictNonCmekServices, then such attempts fail with an error immediately.

It is now possible to use upper-case symbols in the versions of PyPI packages.

If it is not possible to create an environment because of constraints/compute.vmCanIpForward and compute.vmExternalIpAccessorganization policies, then such attempts fail with an error immediately.

Cloud Composer 1.18.9 and 2.0.13 images are available:

• composer-1.18.9-airflow-1.10.15 (default)
• composer-1.18.9-airflow-2.1.4
• composer-1.18.9-airflow-2.2.3
• composer-1.18.9-airflow-2.2.5
• composer-2.0.13-airflow-2.1.4
• composer-2.0.13-airflow-2.2.3
• composer-2.0.13-airflow-2.2.5

(Airflow 2) The default value for the [webserver]worker_refresh_interval Airflow configuration option is changed to 600 seconds.

(Cloud Composer 1) Increased the memory limit for GCSfuse on machine types that have more than 4 GB of memory. This change improves the stability of the syncing process between the environment's bucket and worker pods.

(Available without upgrading) The domain prefix for Private Service Connect subnetwork (connection_subnetwork) is now omitted in environment details.

Cloud Composer 1.18.8 and 2.0.12 images are available:

• composer-1.18.8-airflow-1.10.15 (default)
• composer-1.18.8-airflow-2.1.4
• composer-1.18.8-airflow-2.2.3
• composer-2.0.12-airflow-2.1.4
• composer-2.0.12-airflow-2.2.3

(Cloud Composer 2) When a new environment is created, Cloud Composer checks if the project's network setup allows connections to the Airflow web server domain, *.composer.cloud.google.com. An error is generated if it's not possible to connect to this domain.

Cloud Composer 1.18.7 and 2.0.11 images are available:

• composer-1.18.7-airflow-1.10.15 (default)
• composer-1.18.7-airflow-2.1.4
• composer-1.18.7-airflow-2.2.3
• composer-2.0.11-airflow-2.1.4
• composer-2.0.11-airflow-2.2.3

(Cloud Composer 2) Airflow webserver and worker-scheduler images in multiregional repositories are now tagged with their image version (for example, composer-2.0.10-airflow-2.1.4). This change is gradually backfilled to previous images as well.

(Airflow 1.10.15) Airflow Upgrade Checker updated to version 1.4.0.

Cloud Composer 1.18.6 and 2.0.10 images are available:

• composer-1.18.6-airflow-1.10.15 (default)
• composer-1.18.6-airflow-2.1.4
• composer-1.18.6-airflow-2.2.3
• composer-2.0.10-airflow-2.1.4
• composer-2.0.10-airflow-2.2.3

When syncing objects in an environment bucket to Airflow components, Cloud Composer now reports problems with file names. Such objects no longer cause the syncing process to fail. Objects in the environment's bucket named ".", having names ending in "/", and directories and objects having the same name are reported.

Airflow 2.2.3 packages are updated: added keyrings.google-artifactregistry-auth, updated apache-airflow-providers-google to version 6.6.0.

Cloud Composer 1.18.5 and 2.0.9 images are available:

• composer-2.0.9-airflow-2.2.3
• composer-2.0.9-airflow-2.1.4
• composer-1.18.5-airflow-2.2.3
• composer-1.18.5-airflow-2.1.4
• composer-1.18.5-airflow-1.10.15 (default)

The description of Composer Compute CPU SKUs was changed from "CPU" to "mCPU", to simplify the invoice interpretation. There are no changes in the actual Cloud Composer pricing model.

Improved the validation of custom IP ranges that are specified when an environment is created. The validation is more extensive and redundant error messages were removed.

(New Cloud Composer 1 environments) The minimum disk size for environment nodes is changed from 20 GB to 30 GB.

Cloud Composer 1.18.4 and 2.0.8 images are available:

• composer-2.0.8-airflow-2.2.3
• composer-2.0.8-airflow-2.1.4
• composer-1.18.4-airflow-2.2.3
• composer-1.18.4-airflow-2.1.4
• composer-1.18.4-airflow-1.10.15 (default)

Environment labels are now propagated to the environment's bucket.

Cloud Composer 1.18.3 and 2.0.7 images are available:

• composer-2.0.7-airflow-2.2.3
• composer-2.0.7-airflow-2.1.4
• composer-1.18.3-airflow-2.2.3
• composer-1.18.3-airflow-2.1.4
• composer-1.18.3-airflow-1.10.15 (default)

Improved the handling of errors generated during update and upgrade operations because of an invalid pip.conf configuration file.

(Airflow 1) The apache-airflow-backport-providers-google package is updated to version 2022.2.22.

Cloud Composer 1.18.2 and 2.0.6 images are available:

• composer-2.0.6-airflow-2.2.3
• composer-2.0.6-airflow-2.1.4
• composer-1.18.2-airflow-2.2.3
• composer-1.18.2-airflow-2.1.4
• composer-1.18.2-airflow-1.10.15 (default)

(New environments only) Temporary Pub/Sub topics created during environment operations are now labeled.

(Airflow 2) The google-cloud-datastore package was added to the list of preinstalled packages.

(Airflow 1) The apache-airflow-backport-providers-google package is updated to version 2022.2.11.

(Available without upgrading) Improved the handling of errors in DAG UI for tasks without a set operator.

Cloud Composer 1.18.1 and 2.0.5 images are available:

• composer-2.0.5-airflow-2.2.3
• composer-2.0.5-airflow-2.1.4
• composer-1.18.1-airflow-2.2.3
• composer-1.18.1-airflow-2.1.4
• composer-1.18.1-airflow-1.10.15 (default)

(Cloud Composer 1) Starting from this version, newly created environments use Python 3.8 as the default Python version. Python 3.6 is no longer available. If you upgrade an existing environment to 1.18.0 and later versions, the Python version changes from Python 3.6 to Python 3.8.

Cloud Composer 1.18.0 and 2.0.4 images are available:

• composer-2.0.4-airflow-2.2.3
• composer-2.0.4-airflow-2.1.4
• composer-1.18.0-airflow-2.2.3
• composer-1.18.0-airflow-2.1.4
• composer-1.18.0-airflow-1.10.15 (default)

(Airflow 2.2.3) Support for Deferrable Tasks is not available in Cloud Composer yet.

Cloud Composer 1.17.10 and 2.0.3 images are available:

• composer-2.0.3-airflow-2.2.3
• composer-2.0.3-airflow-2.1.4
• composer-2.0.3-airflow-2.0.2
• composer-1.17.10-airflow-2.2.3
• composer-1.17.10-airflow-2.1.4
• composer-1.17.10-airflow-2.0.2
• composer-1.17.10-airflow-1.10.15 (default)

Cloud Composer 1.17.9 and 2.0.2 images are available:

• composer-2.0.2-airflow-2.1.4
• composer-2.0.2-airflow-2.0.2
• composer-1.17.9-airflow-2.1.4
• composer-1.17.9-airflow-2.0.2
• composer-1.17.9-airflow-1.10.15 (default)

(Available without upgrading) Programmatic calls to Airflow API and Airflow UI in Cloud Composer 2 must now contain OAuth tokens with the https://www.googleapis.com/auth/cloud-platform scope. You can find an example of setting this scope in Make calls to Airflow REST API.

Cloud Composer 1.17.8 and 2.0.1 images are available:

• composer-2.0.1-airflow-2.1.4
• composer-2.0.1-airflow-2.0.2
• composer-1.17.8-airflow-2.1.4
• composer-1.17.8-airflow-2.0.2
• composer-1.17.8-airflow-1.10.15 (default)

Cloud Composer 2.0.0 images are available:

• composer-2.0.0-airflow-2.1.4
• composer-2.0.0-airflow-2.0.2

(Airflow 2.1.4) The apache-airflow-providers-hashicorp package is added to images with Airflow 2.1.4.

(Airflow 2.1.4) Backported the fix for the SerializedDagNotFound: DAG not found in serialized_dag table Airflow bug.

(Airflow 2.1.4) Users with the Admin role can now create users in the Airflow UI.

Cloud Composer 1.17.7 and 2.0.0-preview.7 images are available:

• composer-1.17.7-airflow-1.10.15 (default)
• composer-1.17.7-airflow-2.0.2
• composer-1.17.7-airflow-2.1.4
• composer-2.0.0-preview.7-airflow-2.0.2
• composer-2.0.0-preview.7-airflow-2.1.4

(Available without upgrading) Web server network access control settings can now be configured in Cloud Composer 2 environments.

(Cloud Composer 2) The /dags directory from the environment's bucket is no longer synced to the web server. With this change, the web server startup times are improved.

(Cloud Composer 2) The performance of syncing files to schedulers and workers was improved.

(Airflow 1.10.15) The apache-airflow-backport-providers-google package is updated:

• Wait mechanism for DataprocJobSensor was implemented to handle transient issues when a Job status is not available after creation.
• Support for impersonation_chain in the GKEStartPodOperator.
• New Operators for Dataproc Metastore and Dataproc Serverless.
• Fixed a bug in Dataflow hook when no jobs are returned.
• Google Secret Manager clients are no longer cached.
• Use correct project and location in the on_kill method for Dataflow operators.

New versions of Cloud Composer images:

• composer-1.17.6-airflow-1.10.15 (default)
• composer-1.17.6-airflow-2.0.2
• composer-1.17.6-airflow-2.1.4
• composer-2.0.0-preview.6-airflow-2.0.2
• composer-2.0.0-preview.6-airflow-2.1.4

(Cloud Composer 1) GCSfuse version updated to 37.0. Cloud Composer uses GCSfuse to sync files between the environment bucket and worker pods.

(New environments only) Default values for maintenance windows are from 00:00:00 to 04:00:00 (GMT) on Sunday, Friday, and Saturday every week.

New versions of Cloud Composer images:

composer-1.17.5-airflow-2.1.4 composer-1.17.5-airflow-2.1.2 composer-1.17.5-airflow-2.0.2 composer-1.17.5-airflow-1.10.15 (default) composer-1.17.5-airflow-1.10.12 composer-1.17.5-airflow-1.10.14

The scheduler heartbeat probe no longer checks for the availability of logs in Cloud Logging. This change reduces the chance of false positives.

(Cloud Composer 2 only) The termination grace period for workers is extended from 30 seconds to 30 minutes. This change does not apply to operations that re-create pods (for example, when changing environment variables), so that these operations do not require a long time to complete.

(New Cloud Composer 2 environments only) Access to the Airflow web server in Cloud Composer 2 is now additionally protected by Service Control API checks, for extended access control.

New versions of Cloud Composer images:

• composer-2.0.0-preview.5-airflow-2.1.4
• composer-2.0.0-preview.5-airflow-2.1.2
• composer-2.0.0-preview.5-airflow-2.0.2
• composer-1.17.4-airflow-2.1.4
• composer-1.17.4-airflow-2.1.2
• composer-1.17.4-airflow-2.0.2
• composer-1.17.4-airflow-1.10.15 (default)
• composer-1.17.4-airflow-1.10.14
• composer-1.17.4-airflow-1.10.12

Added the google-cloud-aiplatform package to Cloud Composer images with Airflow versions 2.1.2, 2.0.2, and 1.10.15.

(New environments only) Cloud Composer 2 environments create Autopilot clusters using the Regular release channel. Before this change, the Rapid channel was used.

(New environments only) Shielded Nodes and Secure Boot features are enabled for Cloud Composer 1 environment clusters.

(New environments only) Cloud Composer 1 environment creation no longer fails when the constraints/compute.requireShieldedVm policy is turned on.

New versions of Cloud Composer images:

• composer-1.17.3-airflow-2.1.2
• composer-1.17.3-airflow-2.0.2
• composer-1.17.3-airflow-1.10.15 (default)
• composer-1.17.3-airflow-1.10.14
• composer-1.17.3-airflow-1.10.12
• composer-2.0.0-preview.4-airflow-2.1.2
• composer-2.0.0-preview.4-airflow-2.0.2

Changes in the preinstalled apache-airflow-backport-providers-google package for Airflow 1.10.15:

• Dataflow job operators can be run in async mode.
• Dataflow Hook handles no Job Type.

New versions of Cloud Composer images:

• composer-1.17.2-airflow-2.1.2
• composer-1.17.2-airflow-2.0.2
• composer-1.17.2-airflow-1.10.15 (default)
• composer-1.17.2-airflow-1.10.14
• composer-1.17.2-airflow-1.10.12
• composer-2.0.0-preview.3-airflow-2.1.2 (default)
• composer-2.0.0-preview.3-airflow-2.0.2

Cloud Composer images with Airflow 2 now use Python 3.8.12.

Changes in the preinstalled apache-airflow-backport-providers-google package for Airflow 1.10.15:

• CloudDataFusionStartPipelineOperator can be run in async mode.
• Added a new sensor, DatafusionPipelineStateSensor.
• Fixes for the success_states and pipeline_timeout parameters in CloudDataFusionStartPipelineOperator.

New versions of Cloud Composer images:

• composer-1.17.1-airflow-2.1.2
• composer-1.17.1-airflow-2.0.2
• composer-1.17.1-airflow-1.10.15 (default)
• composer-1.17.1-airflow-1.10.14
• composer-1.17.1-airflow-1.10.12
• composer-2.0.0-preview.2-airflow-2.1.2
• composer-2.0.0-preview.2-airflow-2.0.2

The default value for the visibility-timeout Airflow configuration option, which determines the amount of time after which the task is re-picked by another worker, is increased from 6 hours to 7 days. This change fixes a problem with long-running tasks, when two Airflow workers attempt to write to one log file at the same time, causing tasks to fail with a logging exception.

Airflow has its own system for controlling task health, which is not dependent on visibility-timeout. If required, you can override the value of this configuration option for your environment.

If an Airflow configuration option is blocked in the image version specified for an upgrade operation, and this option has an override in your environment, the upgrade operation is rejected.

New versions of Cloud Composer images:

• composer-1.17.0-airflow-2.1.2
• composer-1.17.0-airflow-2.0.2
• composer-1.17.0-airflow-1.10.15 (default)
• composer-1.17.0-airflow-1.10.14
• composer-1.17.0-airflow-1.10.12
• composer-2.0.0-preview.1-airflow-2.1.2
• composer-2.0.0-preview.1-airflow-2.0.2

Cloud Composer 2 uses the following Cloud Composer images:

• composer-2.0.0-preview.0-airflow-2.1.2
• composer-2.0.0-preview.0-airflow-2.1.1
• composer-2.0.0-preview.0-airflow-2.0.2

(Airflow 2) Cloud Composer now uses a custom authentication backend for authentication in the stable Airflow REST API. The default value of the [api]auth_backend Airflow configuration option is changed to airflow.composer.api.backend.composer_auth. The default role for new users that authenticate through the stable Airflow REST API is defined by the [api]composer_auth_user_registration_role Airflow configuration option, which is set to Op by default.

New versions of Cloud Composer images:

• composer-1.16.16-airflow-1.10.12
• composer-1.16.16-airflow-1.10.14
• composer-1.16.16-airflow-1.10.15 (default)
• composer-1.17.0-preview.12-airflow-2.0.2
• composer-1.17.0-preview.12-airflow-2.1.1
• composer-1.17.0-preview.12-airflow-2.1.2

The default value for the [core]parallelism Airflow configuration option is set to match the maximum worker concurrency in Airflow. Before this change, some tasks could be present in the queue but not executed, and then fail after reaching the timeout.

(Airflow 2.1.1) Airflow configuration options that are deprecated in Airflow 2 are now blocked. In Airflow 2, these options are located in a different configuration section, but Airflow still uses the deprecated location as a fallback.

New versions of Cloud Composer images:

• composer-1.16.15-airflow-1.10.12
• composer-1.16.15-airflow-1.10.14
• composer-1.16.15-airflow-1.10.15 (default)
• composer-1.17.0-preview.11-airflow-2.0.2
• composer-1.17.0-preview.11-airflow-2.1.1

(Airflow 2.1.1) Downgraded colorlog dependency from 5.0.1 to 4.8.0

(Airflow 2.1.1) downgraded google-cloud-memcache dependency from 1.1.0 to 1.0.0

New versions of Cloud Composer images:

• composer-1.16.14-airflow-1.10.12
• composer-1.16.14-airflow-1.10.14
• composer-1.16.14-airflow-1.10.15 (default)
• composer-1.17.0-preview.10-airflow-2.0.2
• composer-1.17.0-preview.10-airflow-2.1.1

(Airflow 2) By default, Airflow performs two retries for a failed task. The default value for the core-default_task_retries Airflow configuration option is 2.

New versions of Cloud Composer images:

• composer-1.16.13-airflow-1.10.12
• composer-1.16.13-airflow-1.10.14
• composer-1.16.13-airflow-1.10.15 (default)
• composer-1.17.0-preview.9-airflow-2.0.2
• composer-1.17.0-preview.9-airflow-2.1.1

New versions of Cloud Composer images:

• composer-1.16.12-airflow-1.10.12
• composer-1.16.12-airflow-1.10.14
• composer-1.16.12-airflow-1.10.15 (default)
• composer-1.17.0-preview.8-airflow-2.0.2
• composer-1.17.0-preview.8-airflow-2.1.1

New versions of Cloud Composer images:

• composer-1.16.11-airflow-1.10.12
• composer-1.16.11-airflow-1.10.14
• composer-1.16.11-airflow-1.10.15 (default)
• composer-1.17.0-preview.7-airflow-2.0.2

New versions of Cloud Composer images:

• composer-1.16.10-airflow-1.10.15 (default)
• composer-1.16.10-airflow-1.10.14
• composer-1.16.10-airflow-1.10.12
• composer-1.17.0-preview.6-airflow-2.0.1
• composer-1.17.0-preview.6-airflow-2.0.2

New versions of Cloud Composer images:

• composer-1.17.0-preview.5-airflow-2.0.1
• composer-1.16.9-airflow-1.10.15 (default)
• composer-1.16.9-airflow-1.10.14
• composer-1.16.9-airflow-1.10.12

New versions of Cloud Composer images:

• composer-1.17.0-preview.4-airflow-2.0.1
• composer-1.16.8-airflow-1.10.15 (default)
• composer-1.16.8-airflow-1.10.14
• composer-1.16.8-airflow-1.10.12

New versions of Cloud Composer images:

• composer-1.17.0-preview.3-airflow-2.0.1
• composer-1.16.7-airflow-1.10.15
• composer-1.16.7-airflow-1.10.14 (default)
• composer-1.16.7-airflow-1.10.12

GCSfuse version was updated to 0.35.1 (latest release). Cloud Composer uses GCSfuse to sync files between the environment buckets and worker pods. The change improves the stability of the syncing process.

(Airflow 2) Preinstalled packages changed. Removed: google_cloud_build==2.0.0, mock==2.0.0, pbr==5.5.1. Downgraded overrides from 3.1.0 to 2.8.0.

For DAG runs that have long-running tasks with certain operators, task level logs are now periodically updated in the Airflow UI. Before this change, logs for such tasks were only available in Airflow UI after the task was completed.

It is now possible to create environments with CMEK encryption in projects with enabled domain restricted sharing. Before the fix, an error related to insufficient Cloud Pub/Sub permissions was generated.

New versions of Cloud Composer images:

• composer-1.17.0-preview.2-airflow-2.0.1
• composer-1.16.6-airflow-1.10.15
• composer-1.16.6-airflow-1.10.14 (default)
• composer-1.16.6-airflow-1.10.12

Increased the timeout for environment upgrade operations to support upgrades for databases up to 16 GB in size. If an upgrade operation times out and the Airflow database size is more than 10 GB, a warning message about the database size is generated.

New versions of Cloud Composer images:

• composer-1.17.0-preview.1-airflow-2.0.1
• composer-1.16.5-airflow-1.10.15
• composer-1.16.5-airflow-1.10.14 (default)
• composer-1.16.5-airflow-1.10.12

Error messages about PyPI package conflicts now contain links to corresponding cluster build logs.

New versions of Cloud Composer images:

• composer-1.17.0-preview.0-airflow-2.0.1
• composer-1.16.4-airflow-1.10.15
• composer-1.16.4-airflow-1.10.14 (default)
• composer-1.16.4-airflow-1.10.12

For new Cloud Composer environments with Airflow 2, SMTP configuration properties for Airflow have new default values:

• smtp_user is set to an empty value by default.
• smtp_password is set to an empty value by default.
• smtp_mail_from is set to a default value used by Airflow.

Improved the error message that is generated when the specified service account does not have enough permissions to run Airflow workloads.

Added troubleshooting information to error messages generated on Airflow web server deployment failures.

GKE clusters of new Cloud Composer environments use Container-Optimized OS with Containerd (cos_containerd) image type.

Kerberos client (krb5-user) package is pre-installed in Cloud Composer container images.

Some environment operations that failed because of networking problems are now retried instead of failing.

New versions of Cloud Composer images:

• composer-1.16.3-airflow-1.10.15
• composer-1.16.3-airflow-1.10.14 (default)
• composer-1.16.3-airflow-1.10.12

New versions of Cloud Composer images:

• composer-1.16.2-airflow-1.10.15
• composer-1.16.2-airflow-1.10.14 (default)
• composer-1.16.2-airflow-1.10.12

When a GKE authorization error occurs during an environment operation, the GKE error message is reported and the operation fails immediately.

New versions of Cloud Composer images:

• composer-1.16.1-airflow-1.10.15
• composer-1.16.1-airflow-1.10.14 (default)
• composer-1.16.1-airflow-1.10.12
• composer-1.16.1-airflow-1.10.10

If an environment's service account does not have required permissions for a requested operation, Cloud Composer generates an actionable error message. The operation fails faster in this case.

New versions of Cloud Composer images:

• composer-1.16.0-airflow-1.10.15
• composer-1.16.0-airflow-1.10.14 (default)
• composer-1.16.0-airflow-1.10.12
• composer-1.16.0-airflow-1.10.10

New versions of Cloud Composer images:

• composer-1.15.2-airflow-1.10.14 (default)
• composer-1.15.2-airflow-1.10.12
• composer-1.15.2-airflow-1.10.10

Irrelevant warnings about asynchronous DAG loading parameters no longer show up in the Airflow logs.

New versions of Cloud Composer images:

• composer-1.15.1-airflow-1.10.14 (default)
• composer-1.15.1-airflow-1.10.12
• composer-1.15.1-airflow-1.10.10

Removed the "@-@" workload info prefixes from Airflow task logs in the Airflow UI and Cloud Logging. This change is available for Airflow 1.10.14.

For Cloud Composer versions 1.13.2 and later, regional base images are used for Airflow web server and worker-scheduler builds. This improves customized image build times.

Invalid resource names in API requests now cause a 4xx response. The invalid resource name is reported in the error message.

New versions of Cloud Composer images:

• composer-1.15.0-airflow-1.10.14
• composer-1.15.0-airflow-1.10.12 (default)
• composer-1.15.0-airflow-1.10.10

When creating new environments, enabling asynchronous DAG loading disables DAG serialization.

Before creating or updating an environment, Cloud Composer checks that required APIs are enabled in a project and that CIDR blocks specified for VPC Native and Private IP are valid. If these requirements are not met, Cloud Composer reports an error and the operation does not start. This change is available only for new Cloud Composer environments.

New versions of Cloud Composer images:

• composer-1.14.5-airflow-1.10.14
• composer-1.14.5-airflow-1.10.12 (default)
• composer-1.14.5-airflow-1.10.10

Improved the logging of Airflow exceptions. Full Python tracebacks for Airflow exceptions are reported and marked as errors in the logs.

New versions of Cloud Composer images:

• composer-1.14.4-airflow-1.10.14
• composer-1.14.4-airflow-1.10.12 (default)
• composer-1.14.4-airflow-1.10.10

When an environment update operation cannot start, an error message that lists possible causes for the error is generated.

Improved the syncing of DAGs and plugins to the Airflow web server. DAG parsing is now less likely to break because of race conditions.

Added FreeTDS system package to Cloud Composer images.

Updated apache-beam package version to 2.24.0 in Airflow 1.10.10 so that Dataflow jobs now correctly create partitioned BigQuery tables. Airflow versions 1.10.12 and 1.10.14 already have apache-beam version 2.27.0 installed.

Upgraded apache-airflow-backport-providers-google package to version 2021.2.5 in Airflow 1.10.12 and 1.10.14. This is potentially a breaking change because the package contains updates of Python Google Cloud libraries. For a list of new operators and for more information about breaking changes, see the 2021.2.5 release notes.

The google-cloud-pubsublite package is installed by default in Composer images for Airflow 1.10.12 and 1.10.14.

Improved the reliability of environment upgrade operations. Added new retrying procedures and enhanced existing ones.

Timeouts for environment upgrade operations are increased.

New versions of Cloud Composer images:

• composer-1.14.2-airflow-1.10.14
• composer-1.14.2-airflow-1.10.12 (default)
• composer-1.14.2-airflow-1.10.10

The default Airflow version for Composer images changes from 1.10.10 to 1.10.12.

New versions of Cloud Composer images:

• composer-1.14.1-airflow-1.10.14
• composer-1.14.1-airflow-1.10.12 (default)
• composer-1.14.1-airflow-1.10.10

• New versions of Cloud Composer images: composer-1.14.0-airflow-1.10.9, composer-1.14.0-airflow-1.10.10, composer-1.14.0-airflow-1.10.12, and composer-1.14.0-airflow-1.10.14. The default is composer-1.14.0-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.

• The maximum number of pods per node in the GKE cluster changes from 100 to 32. This change affects all newly created environments. You can use the max-pods-per-node parameter when creating an environment to increase or decrease the number of pods.

• New versions of Cloud Composer images: composer-1.13.4-airflow-1.10.9, composer-1.13.4-airflow-1.10.10, and composer-1.13.4-airflow-1.10.12. The default is composer-1.13.4-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
• If environment creation succeeds when the requester did not have the iam.serviceAccounts.actAs permission on the service account, Composer will now generate a warning in the audit log entry.
• Error handling for files synchronization between buckets has been improved.

• New versions of Cloud Composer images: composer-1.13.3-airflow-1.10.9, composer-1.13.3-airflow-1.10.10, and composer-1.13.3-airflow-1.10.12. The default is composer-1.13.3-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.13.2-airflow-1.10.9, composer-1.13.2-airflow-1.10.10, and composer-1.13.2-airflow-1.10.12. The default is composer-1.13.2-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.

• Airflow worker pods are now forced to be scheduled on different nodes. Each node runs only a single Airflow worker pod.

• New versions of Cloud Composer images: composer-1.13.0-airflow-1.10.6, composer-1.13.0-airflow-1.10.9, composer-1.13.0-airflow-1.10.10, and composer-1.13.0-airflow-1.10.12. The default is composer-1.13.0-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
• Airflow 1.10.12: api-auth_backend is now set to airflow.api.auth.backend.deny_all by default. To use the experimental REST API, change the api-auth_backend configuration override to airflow.api.auth.backend.default.

• New versions of Cloud Composer images: composer-1.12.5-airflow-1.10.6, composer-1.12.5-airflow-1.10.9, and composer-1.12.5-airflow-1.10.10. The default is composer-1.12.5-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.12.4-airflow-1.10.6, composer-1.12.4-airflow-1.10.9, and composer-1.12.4-airflow-1.10.10. The default is composer-1.12.4-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.12.3-airflow-1.10.6, composer-1.12.3-airflow-1.10.9, and composer-1.12.3-airflow-1.10.10. The default is composer-1.12.3-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.12.2-airflow-1.10.6, composer-1.12.2-airflow-1.10.9, and composer-1.12.2-airflow-1.10.10. The default is composer-1.12.2-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.12.1-airflow-1.10.6, composer-1.12.1-airflow-1.10.9, and composer-1.12.1-airflow-1.10.10. The default is composer-1.12.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.12.0-airflow-1.10.3, composer-1.12.0-airflow-1.10.6, composer-1.12.0-airflow-1.10.9, and composer-1.12.0-airflow-1.10.10. The default is composer-1.12.0-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.11.3-airflow-1.10.3, composer-1.11.3-airflow-1.10.6, and composer-1.11.3-airflow-1.10.9. The default is composer-1.11.3-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
• The max size for serialized DAGs has been increased to 128 MB.
• PyPi dependency conflicts will now cause environment updates to fail by default. This can now be changed by setting the environmental variable IGNORE_PYPI_DEPENDENCY_CONFLICTS=True . This is the default behavior For older environments, but you can set IGNORE_PYPI_DEPENDENCY_CONFLICTS=False to force conflicts to fail.

• New versions of Cloud Composer images: composer-1.11.2-airflow-1.10.3, composer-1.11.2-airflow-1.10.6, and composer-1.11.2-airflow-1.10.9. The default is composer-1.11.2-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.11.1-airflow-1.10.3, composer-1.11.1-airflow-1.10.6, and composer-1.11.1-airflow-1.10.9. The default is composer-1.11.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
• Composer now enforces iam.serviceAccounts.actAs permission checks on the service account specified during Composer environment creation. See Creating environments for details.

• New versions of Cloud Composer images: composer-1.11.0-airflow-1.10.2, composer-1.11.0-airflow-1.10.3, composer-1.11.0-airflow-1.10.6, and composer-1.11.0-airflow-1.10.9. The default is composer-1.11.0-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.10.6-airflow-1.10.2, composer-1.10.6-airflow-1.10.3 and composer-1.10.6-airflow-1.10.6. The default is composer-1.10.6-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

• For Airflow 1.10.6 and later: The Airflow config property [celery] pool is now blocked.

• The [core]sql_alchemy_pool_recycle Airflow setting has been modified to improve SQL connection reliability.

• New versions of Cloud Composer images: composer-1.10.5-airflow-1.10.2, composer-1.10.5-airflow-1.10.3 and composer-1.10.5-airflow-1.10.6. The default is composer-1.10.5-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.10.4-airflow-1.10.2, composer-1.10.4-airflow-1.10.3 and composer-1.10.4-airflow-1.10.6. The default is composer-1.10.4-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
• For Airflow 1.10.6 and later: The Airflow config property [celery] pool is now blocked.

• New versions of Cloud Composer images: composer-1.10.3-airflow-1.10.2, composer-1.10.3-airflow-1.10.3 and composer-1.10.3-airflow-1.10.6. The default is composer-1.10.3-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

• The machine type of the Airflow web server will now be preserved during Composer environment updates, including cases like new PyPi module installations, or adding new environment variables.
• Synchronization of log files between the Airflow scheduler, web server and workers has been improved.
• More useful error messages have been added for Composer environment upgrade failures.
• Future change: Airflow 1.10.6 will become the default Airflow version for Composer environments in an upcoming release.

• New versions of Cloud Composer images: composer-1.10.2-airflow-1.10.2, composer-1.10.2-airflow-1.10.3 and composer-1.10.2-airflow-1.10.6. The default is composer-1.10.2-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.10.0-airflow-1.10.2, composer-1.10.0-airflow-1.10.3 and composer-1.10.0-airflow-1.10.6. The default is composer-1.10.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.9.2-airflow-1.10.1, composer-1.9.2-airflow-1.10.2, composer-1.9.2-airflow-1.10.3 and composer-1.9.2-airflow-1.10.6. The default is composer-1.9.2-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.9.1-airflow-1.10.1, composer-1.9.1-airflow-1.10.2, composer-1.9.1-airflow-1.10.3 and composer-1.9.1-airflow-1.10.6. The default is composer-1.9.1-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.9.0-airflow-1.10.1, composer-1.9.0-airflow-1.10.2, composer-1.9.0-airflow-1.10.3 and composer-1.9.0-airflow-1.10.6. The default is composer-1.9.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.

• Python 3 is now the default version for new Composer environments.
• Airflow 1.9.0 is no longer supported for new Composer environments.

• New versions of Cloud Composer images: composer-1.8.4-airflow-1.9.0, composer-1.8.4-airflow-1.10.1, composer-1.8.4-airflow-1.10.2, and composer-1.8.4-airflow-1.10.3. The default is composer-1.8.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.8.3-airflow-1.9.0, composer-1.8.3-airflow-1.10.1, composer-1.8.3-airflow-1.10.2, and composer-1.8.3-airflow-1.10.3. The default is composer-1.8.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.8.2-airflow-1.9.0, composer-1.8.2-airflow-1.10.1, composer-1.8.2-airflow-1.10.2, and composer-1.8.2-airflow-1.10.3. The default is composer-1.8.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.8.1-airflow-1.9.0, composer-1.8.1-airflow-1.10.1, composer-1.8.1-airflow-1.10.2, and composer-1.8.1-airflow-1.10.3. The default is composer-1.8.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.8.0-airflow-1.9.0, composer-1.8.0-airflow-1.10.1, composer-1.8.0-airflow-1.10.2, and composer-1.8.0-airflow-1.10.3. The default is composer-1.8.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.7.9-airflow-1.9.0, composer-1.7.9-airflow-1.10.1, and composer-1.7.9-airflow-1.10.2. The default is composer-1.7.9-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.7.7-airflow-1.9.0, composer-1.7.7-airflow-1.10.1, and composer-1.7.7-airflow-1.10.2. The default is composer-1.7.7-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.7.6-airflow-1.9.0, composer-1.7.6-airflow-1.10.1, and composer-1.7.6-airflow-1.10.2. The default is composer-1.7.6-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

New versions of Cloud Composer images: composer-1.7.5-airflow-1.9.0, composer-1.7.5-airflow-1.10.1, and composer-1.7.5-airflow-1.10.2. The default is composer-1.7.5-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

New versions of Cloud Composer images: composer-1.7.4-airflow-1.9.0, composer-1.7.4-airflow-1.10.1, and composer-1.7.4-airflow-1.10.2. The default is composer-1.7.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

New versions of Cloud Composer images: composer-1.7.3-airflow-1.9.0, composer-1.7.3-airflow-1.10.1, and composer-1.7.3-airflow-1.10.2. The default is composer-1.7.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.7.2-airflow-1.9.0, composer-1.7.2-airflow-1.10.1, and composer-1.7.2-airflow-1.10.2. The default is composer-1.7.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
• The Airflow configuration webserver-workers is now updatable. Available only in Airflow 1.10.2 and later.

New versions of Cloud Composer images: composer-1.7.1-airflow-1.9.0, composer-1.7.1-airflow-1.10.1, and composer-1.7.1-airflow-1.10.2. The default is composer-1.7.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

• New versions of Cloud Composer images: composer-1.7.0-airflow-1.9.0, composer-1.7.0-airflow-1.10.1, and composer-1.7.0-airflow-1.10.2. The default is composer-1.7.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
• To prevent co-locating workers on the same node, improved Airflow worker pods scheduling.
• When creating a Private IP Cloud Composer environment, the GKE master IP CIDR block no longer needs to be specified.

• Starting from composer-1.7.0-*, Apache Airflow version 1.10.0 is no longer available for environment creation or in-place upgrade.

New versions of Cloud Composer images: composer-1.6.1-airflow-1.9.0, composer-1.6.1-airflow-1.10.0, and composer-1.6.1-airflow-1.10.1. The default is composer-1.6.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

New versions of Cloud Composer images: composer-1.6.0-airflow-1.9.0, composer-1.6.0-airflow-1.10.0, and composer-1.6.0-airflow-1.10.1. The default is composer-1.6.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

New versions of Cloud Composer images: composer-1.4.2-airflow-1.9.0 and composer-1.4.2-airflow-1.10.0.

You can now configure Shared VPC in the Google Cloud Console.

• New versions of Cloud Composer images: composer-1.5.2-airflow-1.9.0, composer-1.5.2-airflow-1.10.0, and composer-1.5.2-airflow-1.10.1. The default is composer-1.5.2-airflow-1.9.0.
• For the Cloud SQL instance for Cloud Composer, the wait_timeout and interactive_timeout are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock.
• Added back the openssh-client package to the default Cloud Composer managed docker images.

• New versions of Cloud Composer images: composer-1.5.1-airflow-1.9.0, composer-1.5.1-airflow-1.10.0, and composer-1.5.1-airflow-1.10.1. The default is composer-1.5.1-airflow-1.9.0.
• For the Cloud SQL instance for Cloud Composer, the wait_timeout and interactive_timeout are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock.
• Added back the openssh-client package to the default Cloud Composer managed docker images.

• New versions of Cloud Composer images: composer-1.5.0-airflow-1.9.0, composer-1.5.0-airflow-1.10.0, and composer-1.5.0-airflow-1.10.1
• Increased the disk size for the Apache Airflow webserver from 10 to 20 GB.

• We now install Airflow dependencies of "mysql" instead of "devel".

  If your DAG uses dependencies that are available only in "devel", add the dependencies to your DAG as PyPI dependencies.

• New versions of Cloud Composer images: composer-1.4.1-airflow-1.9.0 and composer-1.4.1-airflow-1.10.0.
• The Airflow workers restart when a Celery concurrency configuration change is requested so that the change takes effect automatically.
• The logs/ and data/ directories in Cloud Storage are mounted with file mode 755, enabling binaries to be placed in these directories and executed from Airflow.
• The GCP_PROJECT environment variable is set to the Google Cloud Platform project ID in the Airflow webserver.

New versions of Cloud Composer images: composer-1.4.0-airflow-1.9.0 and composer-1.4.0-airflow-1.10.0.

New versions of Cloud Composer images: composer-1.3.0-airflow-1.9.0 and composer-1.3.0-airflow-1.10.0.

• New version of the Cloud Composer image: composer-1.2.0-airflow-1.9.0.
• To reduce potential scheduling delays due to DAG processing, the scheduler restart behavior is now time based (every 600 seconds), not run based.

• New version of the Cloud Composer image: composer-1.1.1-airflow-1.9.0.
• The following Airflow configurations are now updatable: core-dag_concurrency, core-parallelism, core-max_active_runs_per_dag, and scheduler-max_threads.

• New version of the Cloud Composer image: composer-1.1.0-airflow-1.9.0.
• Cloud Composer environments running composer-1.0.0 or later now have GKE cluster auto-upgrade enabled by default.
• Introduced additional liveness monitoring for the Cloud Composer Stackdriver fluentd agent.

• New version of the Cloud Composer image: composer-1.0.0-airflow-1.9.0.
• Cloud Composer environments running composer-1.0.0 or later can now launch Cloud Dataflow jobs in supported Cloud Dataflow regions.
• Added the option to modify some celery configurations.

• New version of the Cloud Composer image: composer-0.5.3-airflow-1.9.0.
• Added requirement to include https://www.googleapis.com/auth/cloud-platform when specifying OAuth scopes during environment creation.
• Improved health checking on Airflow to restart scheduler and worker pods if scheduled tasks are not queued or executed.
• Improved messaging for create and update environment failures.

• New version of the Cloud Composer image: composer-0.5.2-airflow-1.9.0.
• Added OAuth scope configuration on the Google Cloud Platform Console Environment creation page
• Increased the minimum disk size from 10 GB to 20 GB.
• Improved error reporting.
• Improved stability of the CeleryExecutor Redis message backend.

New version of the Cloud Composer image: composer-0.5.1-airflow-1.9.0.