Configure Google Cloud Managed Service for Apache Kafka networking

Managed Service for Apache Kafka deploys clusters in a Google-managed Virtual Private Cloud (VPC). A cluster is exposed in your VPCs by using Private Service Connect (PSC) endpoints. This section explains the details of this architecture.

Before you begin

This section discusses features from several Google Cloud services that are used in Managed Service for Apache Kafka networking.

  • Tenant network: The Google Cloud-managed VPC that hosts Managed Service for Apache Kafka clusters. Each Kafka cluster resides within its own isolated tenant network. This ensures security and isolation.

  • Consumer network: The VPC where you run your Kafka clients. Note that this is not relevant to the clients that connect to the Google APIs since these are not served on a VPC.

  • PSC: PSC connects the tenant network to the consumer network. It lets you create IP addresses for the Kafka cluster, including the bootstrap address and brokers, in a specific subnet.

  • Cloud DNS: The Google Cloud service creates DNS entries associated with PSC IP addresses. These entries are registered in Cloud DNS using a split-horizon DNS zone. The registered URLs are a property of the cluster, rather than a VPC.

Required permissions

To enable cross-project access using PSC, you must grant specific permissions to the Google-managed service agent, service-<project_number>@gcp-sa-managedkafka.iam.gserviceaccount.com. Replace <project_number> with the project number of the project that contains the Managed Service for Apache Kafka cluster. This service account is automatically created for each project and is responsible for managing your clusters.

Assign the Managed Service for Apache Kafka service agent role (roles/managedkafka.serviceAgent) to this service account in each project where you want to create PSC endpoints. This role includes the specific permissions needed for PSC configuration.

For more information on how to grant a role to a service account, see Manage access to service accounts.

Networking workflow

The following is a brief overview of the steps that you need to take to set up networking for your clusters:

  1. Determine the projects where you want your Kafka clients to access the Managed Service for Apache Kafka cluster.

  2. In each of the projects that contains the VPC network, grant the roles/managedkafka.serviceAgent role to the service-<project_number>@gcp-sa-managedkafka.iam.gserviceaccount.com service agent associated with the project that hosts the Managed Service for Apache Kafka cluster.

  3. Choose a subnet within each VPC to assign IP addresses for your cluster. You'll need one IP address for the bootstrap address and one for each broker. The smallest Managed Service for Apache Kafka cluster contains 3 brokers.

  4. Set up your cluster to connect to the chosen subnets.

    For more information about creating the cluster, see Create a Managed Service for Apache Kafka cluster.

  5. Your applications within those VPC networks can now connect to your clusters using the PSC endpoints.

Network architecture of a cluster

  • A Kafka cluster spans a tenant network and one or more VPC networks.

  • A single bootstrap IP address and URL is used for the cluster. This bootstrap address corresponds to a load balancer connected to all the brokers in the cluster. Each broker can also act as a bootstrap server.

  • Within each VPC, a single PSC endpoint is provisioned for the bootstrap address and one for each broker. The URL for the bootstrap address is the same across the consumer VPCs to which a cluster is connected.

  • PSC lets you associate a unique IP address for each Kafka server. This IP address is local to the consumer VPC.

  • Clients can connect to Kafka brokers using either internal IP addresses or DNS names. However, using DNS names is recommended. These names are registered automatically in every VPC to which a Kafka cluster is connected. The bootstrap server is typically accessible at bootstrap.CLUSTER_NAME.LOCATION.managedkafka.PROJECT_ID.cloud.goog, while brokers are usually registered as broker-N.CLUSTER_NAME.LOCATION.managedkafka.PROJECT_ID.cloud.goog:9092. The bootstrap address and its port number are also available as a property of the cluster.

  • The bootstrap address is used by clients to retrieve broker URLs that are resolved to IP addresses local to each VPC. You can, however, find actual broker IP addresses and URLs in Cloud DNS.

The following image shows a sample architecture of a Kafka cluster network.

A Managed Service for Apache Kafka network
Figure 1. A Managed Service for Apache Kafka network

  • In this example, the cluster has three brokers and the cluster is in the tenant VPC.

  • Brokers communicate with clients over the default Managed Service for Apache Kafka port (9092) and have unique IP addresses. In this example, the three brokers have IP addresses 10.128.10.1, 10.128.10.2, and 10.128.10.3 respectively.

  • All three brokers connect to the bootstrap load balancer. This ensures high availability and regional fault tolerance, as the bootstrap address is not confined to a single broker or zone.

What's next?