Configure Google Cloud Managed Service for Apache Kafka networking

You can connect to your Google Cloud Managed Service for Apache Kafka cluster from any Virtual Private Cloud (VPC) in your Google Cloud project. The service automatically handles all the networking configuration so that you can access your cluster.

When you create a Managed Service for Apache Kafka cluster, you must select at least one subnet in your VPC. This subnet provides the IP addresses that your cluster uses to communicate with other resources in your VPC. By selecting a subnet, you make the cluster accessible within your VPC network.

For those interested in the underlying implementation, Managed Service for Apache Kafka uses Private Service Connect (PSC) to expose your cluster in your VPCs. This means you can securely connect to your cluster without managing any networking infrastructure.

While the service automates most of the networking setup, understanding some implementation details can be helpful. This section explains the details of the networking architecture used in Managed Service for Apache Kafka.

Before you begin

This section discusses features from several Google Cloud services that are used in Managed Service for Apache Kafka networking.

  • Tenant network: The Google Cloud-managed VPC that hosts Managed Service for Apache Kafka clusters. Each Kafka cluster resides within its own isolated tenant network. This ensures security and isolation.

  • Consumer network: The VPC where you run your Kafka clients. Note that this is not relevant to the clients that connect to the Google APIs since these are not served on a VPC.

  • PSC: PSC connects the tenant network to the consumer network. It lets you create IP addresses for the Kafka cluster, including the bootstrap address and brokers, in a specific subnet.

  • Cloud DNS: The Google Cloud service creates DNS entries associated with PSC IP addresses. These entries are registered in Cloud DNS using a split-horizon DNS zone. The registered URLs are a property of the cluster, rather than a VPC.

Required permissions

To enable cross-project access using PSC, you must grant specific permissions to the Google-managed service agent, service-<project_number>@gcp-sa-managedkafka.iam.gserviceaccount.com. Replace <project_number> with the project number of the project that contains the Managed Service for Apache Kafka cluster. This service account is automatically created for your project when you create your first Managed Service for Apache Kafka cluster and is responsible for managing your clusters.

Assign the Managed Service for Apache Kafka service agent role (roles/managedkafka.serviceAgent) to this service account in each project where you want to create PSC endpoints. This role includes the specific permissions needed for PSC configuration.

gcloud projects add-iam-policy-binding PSC_PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-managedkafka.iam.gserviceaccount.com \
    --role=roles/managedkafka.serviceAgent

Replace the following:

  • PSC_PROJECT_ID: The project where the PSC endpoints will be created. This is the parent project of the VPC network where you run your Kafka clients.

  • PROJECT_NUMBER: The project number of the Managed Service for Apache Kafka cluster.

Networking workflow

The following is a brief overview of the steps that you need to take to set up networking for your clusters:

  1. Determine the projects where you want your Kafka clients to access the Managed Service for Apache Kafka cluster.

  2. In each of the projects that contains the VPC network, grant the roles/managedkafka.serviceAgent role to the service-<project_number>@gcp-sa-managedkafka.iam.gserviceaccount.com service agent associated with the project that hosts the Managed Service for Apache Kafka cluster.

  3. Choose a subnet within each VPC to assign IP addresses for your cluster. You'll need one IP address for the bootstrap address and one for each broker. The smallest Managed Service for Apache Kafka cluster contains 3 brokers.

  4. Set up your cluster to connect to the chosen subnets.

    For more information about creating the cluster, see Create a Managed Service for Apache Kafka cluster.

  5. Your applications within those VPC networks can now connect to your clusters using the PSC endpoints.

Network architecture of a cluster

  • A Kafka cluster spans a tenant network and one or more VPC networks.

  • A single bootstrap IP address and URL is used for the cluster. This bootstrap address corresponds to a load balancer connected to all the brokers in the cluster. Each broker can also act as a bootstrap server.

  • Within each VPC, a single PSC endpoint is provisioned for the bootstrap address and one for each broker. The URL for the bootstrap address is the same across the consumer VPCs to which a cluster is connected.

  • PSC lets you associate a unique IP address for each Kafka server. This IP address is local to the consumer VPC.

  • Clients can connect to Kafka brokers using either internal IP addresses or DNS names. However, using DNS names is recommended. These names are registered automatically in every VPC to which a Kafka cluster is connected. The bootstrap address is typically accessible at bootstrap.CLUSTER_NAME.LOCATION.managedkafka.PROJECT_ID.cloud.goog:9092, while brokers are usually registered as broker-N.CLUSTER_NAME.LOCATION.managedkafka.PROJECT_ID.cloud.goog:9092. The bootstrap address and its port number are also available as a property of the cluster.

  • The bootstrap address is used by clients to retrieve broker URLs that are resolved to IP addresses local to each VPC. You can, however, find actual broker IP addresses and URLs in Cloud DNS.

The following image shows a sample architecture of a Kafka cluster network.

A Managed Service for Apache Kafka network
Figure 1. A Managed Service for Apache Kafka network

  • In this example, the cluster has three brokers and the cluster is in the tenant VPC.

  • Brokers communicate with clients over the default Managed Service for Apache Kafka port (9092) and have unique IP addresses. In this example, the three brokers have IP addresses 10.128.10.2, 10.128.10.3, and 10.128.10.4 respectively.

  • All three brokers connect to the bootstrap load balancer. This ensures high availability and regional fault tolerance, as the bootstrap address is not confined to a single broker or zone.

Subnet sizing

You must ensure that the subnets provided to Managed Service for Apache Kafka have sufficient IP space available for the Private Service Connect endpoints.

The minimum cluster size for Managed Service for Apache Kafka consists of 3 brokers and a bootstrap address, so you need to ensure there are at least 4 usable IP addresses on each subnet connected to the Managed Service for Apache Kafka cluster.

For clusters with 45 vCPUs or more, you'll have 1 broker for every 15 vCPUs. To calculate the minimum number of IP addresses needed for each subnet for such a cluster, follow these steps:

  1. Divide the number of vCPUs by 15.
  2. Round up to the nearest whole number.
  3. Add 1 to the result, to account for the bootstrap address.

The result is the minimum number of IP addresses needed for each subnet. For example, a cluster with 60 vCPUs would require (60/15 + 1) or 5 usable IP addresses.

When calculating the number of IP addresses needed, you should perform the calculation against the maximum expected future size of the cluster.

Google may change the broker to vCPU ratio in the future, so you should provision enough IP addresses to accommodate any future changes. We recommended that you allocate 3x the number of IP addresses calculated in the previous step.

Privately used public IP ranges

You can connect Managed Service for Apache Kafka clusters to subnets that use the non-RFC 1918 address space without additional configuration. Such ranges are called privately used public IP (PUPI) ranges.

The PUPI subnets provided must use a valid IPv4 range that is not a prohibited IPv4 subnet range.

Troubleshooting consumer VPC network configuration

If the service is unable to configure the consumer VPC network for access to Managed Service for Apache Kafka clusters, it logs messages like Managed Service for Apache Kafka failed to set up networking in VPC subnet to the cluster project.

The following is a list of suggested steps that you can take when you have clients failing to connect to Managed Service for Apache Kafka clusters:

  • You must enable the Compute Engine and Cloud DNS APIs in the parent project of the consumer VPC network.

  • If the Managed Service for Apache Kafka cluster and consumer VPC network reside in different projects, configure the required permissions.

  • You must ensure no organization policy constraints prevent the service from creating the necessary resources in the consumer VPC network's project.

  • The bootstrap address must be bootstrap.CLUSTER_NAME.LOCATION.managedkafka.CLUSTER_PROJECT_ID.cloud.goog:9092.

  • You must ensure that the Kafka clients are running in a VPC network that has been configured to access the Managed Service for Apache Kafka cluster.

  • If you are running the Kafka client on a computer or laptop, you can set up a Compute Engine instance to use as a proxy to access to the Managed Service for Apache Kafka cluster.

What's next?