Configure access to the internet when installing PyPI packages

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

This page explains how to disable or enable internet access when installing PyPI packages in your environment.

For information about installing packages and configuring custom sources (such as a repository in your VPC network), see Install Python dependencies.

If you want to enable access to your VPC network from an environment, see Connect an environment to a VPC network. For example, you can configure your VPC network so that your environment can access the internet through it.

If you want to change your environment to Private IP, see Change environment networking type (Private or Public IP).

About the internet access when installing PyPI packages

  • By default, your Cloud Composer 3 environment can access the internet when installing PyPI packages.

  • Packages can now be installed from both public and private sources at the same time. This wasn't possible in Cloud Composer 2.

  • This environment configuration option is independent of the environment's networking type (Private or Public IP). For example, you can install packages from Python Package Index in a Private IP environment.

  • You can set this configuration option when you create an environment and it's possible to change this option for an existing environment.

Disable or enable internet access when installing PyPI packages

Console

  1. In the Google Cloud console, go to the Environments page.

    Go to Environments

  2. In the list of environments, click the name of your environment. The Environment details page opens.

  3. Go to the Environment configuration tab.

  4. In the Network configuration section, find the Access to public PyPI package repositories item and click Edit.

  5. In the Access to public PyPI package repositories dialog:

    • To enable internet access, select Allow installation of packages from public internet repositories.

    • To disable internet access, select Don't allow installation of packages from public internet repositories.

  6. Click Save.

gcloud

The following Google Cloud CLI arguments specify internet access configuration when installing PyPI packages:

  • --disable-private-builds-only enables internet access when installing packages (default).

  • --enable-private-builds-only disables internet access when installing packages.

To enable internet access when installing packages, run the following Google Cloud CLI command:

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --disable-private-builds-only

To disable internet access when installing packages, run the following Google Cloud CLI command:

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --enable-private-builds-only

Replace the following:

  • ENVIRONMENT_NAME: the name of the environment
  • LOCATION: the region where the environment is located

Example (disable internet access):

gcloud beta composer environments update example-environment \
  --location us-central1 \
  --enable-private-builds-only

API

  1. Create an environments.patch API request.

  2. In this request:

    1. In the updateMask parameter, specify the config.private_environment_config.enable_private_builds_only mask.

    2. In the request body, in the enablePrivateBuildsOnly field, set the required value:

      • false: enable internet access when installing PyPI packages (default).
      • true: disable internet access when installing PyPI packages.

Example (disable internet access):

// PATCH https://composer.googleapis.com/v1beta1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.private_environment_config.enable_private_builds_only

"config": {
  "privateEnvironmentConfig": {
    "enablePrivateBuildsOnly": true
  }
}

Terraform

The enable_private_builds_only field in the config block specifies internet access configuration when installing PyPI packages:

  • false or omitted: enables internet access when installing packages (default).

  • true: disables internet access when installing packages.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    enable_private_builds_only = PRIVATE_BUILDS_MODE
  }
}

Replace the following:

  • ENVIRONMENT_NAME: the name of your environment.
  • LOCATION: the region where the environment is located.
  • PRIVATE_BUILDS_MODE: whether the internet access is disabled when installing PyPI packages.

Example (disable internet access):

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    enable_private_builds_only = true

    ... other configuration parameters

  }

What's next