VPC Service Controls with Vertex AI

VPC Service Controls can help you mitigate the risk of data exfiltration from Vertex AI. Use VPC Service Controls to create a service perimeter that protects the resources and data that you specify. For example, when you use VPC Service Controls to protect Vertex AI, the following artifacts can't leave your service perimeter:

  • Training data for an AutoML model or custom model
  • Models that you created
  • Models that you searched by using Neural Architecture Search
  • Requests for online predictions
  • Results from a batch prediction request
  • Gemini models

Controlling access to Google APIs

Vertex AI APIs, as outlined in Accessing Vertex AI from on-premises and multicloud, encompass a range of accessibility options, including public internet, Private Service Connect for Google APIs, and Private Google Access.

Public access

By default, these public APIs are reachable from the internet; however, IAM permissions are required for use. While features like Private Service Connect and Private Google Access facilitate private communication over hybrid network architectures, they don't eliminate public internet accessibility for Vertex AI APIs.

To establish granular control over API access and explicitly restrict public internet exposure, the implementation of VPC Service Controls becomes essential. This security layer empowers administrators to define and enforce access policies, selectively enabling either public or private connectivity to Google APIs to satisfy organizational requirements.

Private access

Organizations that need to restrict public Google APIs to private access can use VPC Service Controls in combination with Private Service Connect Google APIs (VPC Service Controls bundle) or Private Google Access. When deployed over hybrid networking and within Google Cloud, both options enable private access to Google APIs from on-premises. However, Private Service Connect also offers flexibility in defining a custom IP address and DNS endpoint name.

As a best practice, use the restricted Virtual IP (VIP) with Private Service Connect Google APIs or Private Google Access to provide a private network route for requests to Google Cloud services without exposing the requests to the internet. The restricted VIP supports all of the APIs that VPC Service Controls can protect that require considerations for on-premises and VPC networks. Following are some examples:

Controlling API access through private services access

The following Vertex AI APIs deployed with private services access require additional networking configuration when implemented in an environment protected with VPC Service Controls:

  • Vector Search (index query)
  • Custom training (data plane)
  • Vertex AI Pipelines
  • Private online prediction endpoints

For example, Vertex AI Pipelines is a Google-managed (producer) service, deployed in a single-tenant project and VPC network with the ability to scale supported services based on consumer requirements. Communication between the producer and consumer networks is established with VPC Network Peering, except for internet egress, which is routed through the producer network.

In the producer network a default route exists that allows for internet egress, in addition to unrestricted access to Google APIs. Updating the producer network to support the restricted VIP requires enabling VPC Service Controls for peerings, which performs the following actions on all supported services deployed in your service networking producer network:

  • Removes the IPv4 default route (destination 0.0.0.0/0, next hop default internet gateway).
  • Creates Cloud DNS-managed private zones and authorizes those zones for the service producer VPC network. The zones include googleapis.com, pkg.dev, gcr.io, and other necessary domains or host names for Google APIs and services that are compatible with VPC Service Controls.
  • Record data in the zones resolves all host names to 199.36.153.4, 199.36.153.5, 199.36.153.6, and 199.36.153.7.

An alternate method for removing the default route from the producer network without impacting existing Google-managed services is to use HA VPN over Cloud Interconnect consisting of the following steps:

  1. Deploy a services VPC network in addition with HA VPN to the consumer VPC network.
  2. Deploy Google-managed services in the services VPC network.
  3. Enable VPC Service Controls for peerings.
  4. Advertise the private services access subnet as a custom route advertisement from the Cloud Router if the managed services requires on-premises reachability.
  5. Update the service networking VPC network peering with the export custom routes option.

Service perimeter creation

For an overview of creating a service perimeter, see Creating a service perimeter in the VPC Service Controls documentation.

Adding restricted services to your perimeter

When establishing a service perimeter, we recommend that you include all restricted services as a security best practice. This comprehensive approach helps to minimize potential vulnerabilities and unauthorized access. However, there might be scenarios where your organization has specific requirements focused on safeguarding Vertex AI and its interconnected APIs. In such cases, you have the flexibility to select and include only the specific Vertex AI APIs that are essential for your operations.

Vertex AI APIs that you can incorporate into your service perimeter include the following:

  • Vertex AI API supports the following services and features:
    • Batch prediction
    • Datasets
    • Vertex AI Feature Store (Bigtable online serving)
    • Vertex AI Feature Store (optimized online serving)
    • Generative AI on Vertex AI (Gemini)
    • Vertex AI Model Registry
    • Online prediction
    • Vector Search (index creation)
    • Vector Search (index query)
    • Custom training (control plane)
    • Custom training (data plane)
    • Vertex AI Pipelines
    • Private online prediction endpoints
    • Colab Enterprise
  • Notebooks API supports the following service:
    • Vertex AI Workbench

VPC Service Controls support for Generative AI tuning pipelines

VPC Service Controls support is provided in the tuning pipeline of the following models:

  • text-bison for PaLM 2
  • BERT
  • T5
  • The textembedding-gecko family of models.

Using VPC Service Controls with Vertex AI Pipelines

The service perimeter blocks access from Vertex AI to third-party APIs and services on the internet. If you're using Google Cloud Pipeline Components or creating your own custom pipeline components for use with Vertex AI Pipelines, you can't install PyPI dependencies from the public Python Package Index (PyPI) registry. Instead, you must do one of the following:

Use custom containers

As a production software best practice, component authors should use containerized Python components and build the dependencies into their container image, so no live installation is required during a pipeline run. The Kubeflow Pipelines SDK offers a way to containerize your Python code. For more information, see Containerized Python Components.

Install packages from an Artifact Registry repository

Alternatively, you can create an Artifact Registry repository in your project, store Python packages in it, and configure your Vertex AI environment to install from it as outlined in this section. For more information, see Manage Python packages.

Configure roles and permissions

  1. The service account for your Vertex AI environment must have the iam.serviceAccountUser role.

  2. If you install custom PyPI packages from a repository in your project's network, and this repository does not have a public IP address:

    1. Assign permissions to access this repository to the environment's service account.

    2. Make sure that connectivity to this repository is configured in your project.

Create the repository

  1. Create an Artifact Registry repository in VPC mode in your project.
  2. Store the required Python packages in the repository.

Configure the Vertex AI environment to install from the repository

To install custom PyPI packages from one or more Artifact Registry repositories, make a call similar to the following to @dsl.component:

@dsl.component(packages_to_install=["tensorflow"],
pip_index_urls=['http://myprivaterepo.com/simple', 'http://pypi.org/simple'],)
def hello_world(text: str) -> str:
    import my_package
    import tensorflow

    return my_package.hello_world(text)

Limitations

The following limitations apply when you use VPC Service Controls:

  • For data labeling, you must add labelers' IP addresses to an access level.
  • For Google Cloud Pipeline Components, the components launch containers that check their base image for all requirements. The KFP package, as well as any packages listed in the packages_to_install argument are the requirements for a container. If any specified requirements aren't already present in the base image (either provided or custom), the component attempts to download them from the Python Package Index (PyPI). Because the service perimeter blocks access from Vertex AI to third-party APIs and services on the internet, the download fails with Connection to pypi.org timed out. For ways to avoid this error, see Using VPC Service Controls with Vertex AI Pipelines.
  • When using VPC Service Controls with custom kernels in Vertex AI Workbench, you must instead configure DNS peering to send requests for *.notebooks.googleusercontent.com to the subnet 199.36.153.8/30 (private.googleapis.com) instead of 199.36.153.4/30 (restricted.googleapis.com).

What's next