Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3
This page is a companion to the main page about creating environments. It demonstrates how to set up a Cloud Composer environment and a user-managed service account for this environment in an existing Google Cloud project with Terraform. You can use this page as a start, then add more configuration parameters for your environment, as needed.
Before you begin
This guide assumes that you have a Google Cloud project with configured billing:
- You can use an existing project.
- You can create a new project using Google Cloud console, Google Cloud CLI, API, or a Python client library.
- You can create and manage your project using Terraform. For more
information, see Terraform documentation for the
google_project
resource.
Authenticate with Google Cloud
To authenticate with Google Cloud, run:
gcloud auth application-default login
For more information about this command, see
gcloud auth application-default
.
Configure the Google provider in Terraform
Specify your existing project ID and a default region for resources. Your Cloud Composer environment uses this region.
provider "google-beta" {
project = "example-project"
region = "us-central1"
}
Enable the Cloud Composer API
Enable the Cloud Composer API in your project:
resource "google_project_service" "composer_api" {
provider = google-beta
project = "example-project"
service = "composer.googleapis.com"
// Disabling Cloud Composer API might irreversibly break all other
// environments in your project.
// This parameter prevents automatic disabling
// of the API when the resource is destroyed.
// We recommend to disable the API only after all environments are deleted.
disable_on_destroy = false
// this flag is introduced in 5.39.0 version of Terraform. If set to true it will
//prevent you from disabling composer_api through Terraform if any environment was
//there in the last 30 days
check_if_service_has_usage_on_destroy = true
}
Create a custom service account in your project
By default, Cloud Composer environments use the default Compute Engine account. This guide demonstrates another approach, by creating a new service account, which has all required permissions to run a Cloud Composer environment.
Define a custom service account with the following roles and permissions. For more information about permissions for environment's service accounts, see Access control with IAM.
resource "google_service_account" "custom_service_account" {
provider = google-beta
account_id = "custom-service-account"
display_name = "Example Custom Service Account"
}
resource "google_project_iam_member" "custom_service_account" {
provider = google-beta
project = "example-project"
member = format("serviceAccount:%s", google_service_account.custom_service_account.email)
// Role for Public IP environments
role = "roles/composer.worker"
}
Grant required permissions to Cloud Composer service account
Add a new role binding to your environment's service account allow policy.
You add Cloud Composer Service Agent account as a new principal on your environment's service account and grant the Cloud Composer v2 API Service Agent Extension role to it.
If you are not using Terraform to define your project's allow policy, do not use the following example. Instead, add this binding using other methods.
resource "google_service_account_iam_member" "custom_service_account" {
provider = google-beta
service_account_id = google_service_account.custom_service_account.name
role = "roles/composer.ServiceAgentV2Ext"
member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
}
Create an environment
Create your environment using Terraform.
The example demonstrates how to create an environment that uses a custom service account. You can add more parameters that define other configuration parameters of your environment, such as custom scale and performance parameters, or additional PyPI packages.
For more information about other parameters, see Create environments.
resource "google_composer_environment" "example_environment" {
provider = google-beta
name = "example-environment"
config {
software_config {
image_version = "composer-2.9.5-airflow-2.9.1"
}
node_config {
service_account = google_service_account.custom_service_account.email
}
}
}
Full Terraform script (default parameters)
provider "google-beta" {
project = "example-project"
region = "us-central1"
}
resource "google_project_service" "composer_api" {
provider = google-beta
project = "example-project"
service = "composer.googleapis.com"
// Disabling Cloud Composer API might irreversibly break all other
// environments in your project.
disable_on_destroy = false
// this flag is introduced in 5.39.0 version of Terraform. If set to true it will
//prevent you from disabling composer_api through Terraform if any environment was
//there in the last 30 days
check_if_service_has_usage_on_destroy = true
}
resource "google_project_iam_member" "service_agent_role" {
provider = google-beta
project = "example-project"
member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
role = "roles/composer.ServiceAgentV2Ext"
}
resource "google_composer_environment" "example_environment" {
provider = google-beta
name = "example-environment"
config {
// Add your environment configuration here
software_config {
image_version = "composer-2.9.5-airflow-2.9.1"
}
}
}
Full Terraform script (custom service account)
provider "google-beta" {
project = "example-project"
region = "us-central1"
}
resource "google_project_service" "composer_api" {
provider = google-beta
project = "example-project"
service = "composer.googleapis.com"
// Disabling Cloud Composer API might irreversibly break all other
// environments in your project.
disable_on_destroy = false
// this flag is introduced in 5.39.0 version of Terraform. If set to true it will
//prevent you from disabling composer_api through Terraform if any environment was
//there in the last 30 days
check_if_service_has_usage_on_destroy = true
}
resource "google_service_account" "custom_service_account" {
provider = google-beta
account_id = "custom-service-account"
display_name = "Example Custom Service Account"
}
resource "google_project_iam_member" "custom_service_account" {
provider = google-beta
project = "example-project"
member = format("serviceAccount:%s", google_service_account.custom_service_account.email)
// Role for Public IP environments
role = "roles/composer.worker"
}
resource "google_service_account_iam_member" "custom_service_account" {
provider = google-beta
service_account_id = google_service_account.custom_service_account.name
role = "roles/composer.ServiceAgentV2Ext"
member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
}
resource "google_composer_environment" "example_environment" {
provider = google-beta
name = "example-environment"
config {
software_config {
image_version = "composer-2.9.5-airflow-2.9.1"
}
node_config {
service_account = google_service_account.custom_service_account.email
}
}
}
What's next
See other documentation pages for information about configuring your environment with Terraform. For example:
- Create environments
- Override Airflow configuration options
- Set environment variables
- Install Python dependencies
- Scale environments