Change environment networking type (Private or Public IP)

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains the difference between Private IP and Public IP environment networking types in Cloud Composer 3 and provides instructions for switching the networking type of your environment.

If you want to disable or enable internet access only when installing PyPI packages, see Configure internet access when installing PyPI packages.

If you want to enable access to your VPC network from your environment, see Connect an environment to a VPC network.

About environment networking types

Cloud Composer 3 uses two environment networking types:

• Public IP networking:

  • Airflow components of the environment can access the internet. This is the default networking type.

  • Airflow components in Public IP environments establish outbound connections from automatically allocated public IP addresses and ports. If you want your Public IP environments to use predetermined IP addresses and ports, you can do so through connecting a VPC network to your environment and switching it to Private IP. In this case, Cloud Composer routes all traffic except traffic to Google services through this network.

• Private IP networking:

  • Airflow components of the environment don't have access to the internet.

  • Private IP environments configure Private Google Access through the private.googleapis.com range, which enables access to Google APIs, services, and domains supported by this range.

    For more information and the list of services and domains available through private.googleapis.com, see Network configuration in the Virtual Private Cloud documentation.

  • Private IP environments with VPC Service Controls configure Private Google Access through the restricted.googleapis.com range, which enables access to Google APIs, services, and domains supported by this range.

    For more information and the list of services and domains available through restricted.googleapis.com, see Network configuration in the Virtual Private Cloud documentation.

In addition to two networking types, you can enable or disable access to a custom VPC network for any type of environment. Depending on how you configure your VPC network, a Private IP environment can gain access the internet through you VPC network.

Cloud Composer 2 networking compared to Cloud Composer 3

In Cloud Composer 3, Private IP environments require no configuration.

The following Cloud Composer 2 networking features are no longer relevant in Cloud Composer 3:

• Configuring Private IP networking. You don't need to specify IP ranges, networks, or configure connectivity and firewall rules.

• Configuring Private Service Connect. You don't need to set ranges for Private Service Connect in Cloud Composer 3.

• Using privately used public IP ranges. This feature provided an option to extend the available IP ranges, which are not required in Cloud Composer 3.

• Using the IP Masquerade agent. You don't need to configure cluster connectivity in Cloud Composer 3.

• Configuring authorized networks. It is not possible to access the environment's cluster in Cloud Composer 3.

Change environment networking type

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --enable-private-environment

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --disable-private-environment

gcloud beta composer environments update example-environment \
  --location us-central1 \
  --enable-private-environment

gcloud beta composer environments update example-environment \
  --location us-central1 \
  --disable-private-environment

// PATCH https://composer.googleapis.com/v1beta1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.private_environment_config.enable_private_environment

"config": {
  "privateEnvironmentConfig": {
    "enablePrivateEnvironment": true
  }
}

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {

    enable_private_environment = PRIVATE_IP_STATUS

  }
}

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    enable_private_environment = true

    ... other configuration parameters
  }
}

Configure proxy server variables

You can set http_proxy and https_proxy environment variables in your environment. These standard Linux variables are used by web clients that run in containers of your environment's cluster to route traffic through the specified proxies.

The NO_PROXY variable by default is set to a list of Google domains and localhost so that they are excluded from proxying: .google.com,.googleapis.com,metadata.google.internal,localhost. This configuration makes it possible to create an environment with set http_proxy and https_proxy environment variables in cases when the proxy isn't configured to handle traffic to Google services.

What's next

• Configure VPC Service Controls