List your Apache Kafka for BigQuery clusters

To list your clusters, you can use the Google Cloud console, the Google Cloud CLI, the client library, or the Apache Kafka for BigQuery API. You can't use the open source Apache Kafka API to list a cluster.

Required roles and permissions to list your clusters

To get the permissions that you need to list your clusters, ask your administrator to grant you the Managed Kafka Viewer (roles/managedkafka.viewer) IAM role on your project. For more information about granting roles, see Manage access.

This predefined role contains the permissions required to list your clusters. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to list your clusters:

  • List clusters: managedkafka.clusters.list
  • Get cluster details: managedkafka.clusters.get

You might also be able to get these permissions with custom roles or other predefined roles.

For more information about the Managed Kafka Viewer role, see Apache Kafka for BigQuery predefined roles.

List your clusters

To list all your clusters in a project, follow these steps:

Console

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. The clusters you created in a project are listed. The page lets you view the following cluster properties:
    • Cluster name: The unique identifier for your Apache Kafka for BigQuery cluster. You can use it to reference the cluster in various operations.
    • Cluster state: Indicates the current operational status of the cluster, such as Active.
    • Region: The geographical location where your Managed Kafka cluster is hosted.
    • Memory: The total amount of memory allocated to the cluster. This determines the capacity for handling message traffic and storage.
    • vCPU: The number of vCPUs assigned to the cluster. This affects the cluster's processing power and ability to handle concurrent operations.
    • Labels: Key-value pairs that you can attach to the cluster for organization, filtering, and automation purposes.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Run the gcloud beta managed-kafka clusters list command:

    gcloud beta managed-kafka clusters list --location=LOCATION \
    --limit=LIMIT

    Replace the following:

    • LOCATION: The location of the cluster.

    • LIMIT: The maximum number of clusters to list.

Go

import (
	"context"
	"fmt"
	"io"

	"cloud.google.com/go/managedkafka/apiv1/managedkafkapb"
	"google.golang.org/api/iterator"
	"google.golang.org/api/option"

	managedkafka "cloud.google.com/go/managedkafka/apiv1"
)

func listClusters(w io.Writer, projectID, region string, opts ...option.ClientOption) error {
	// projectID := "my-project-id"
	// region := "us-central1"
	ctx := context.Background()
	client, err := managedkafka.NewClient(ctx, opts...)
	if err != nil {
		return fmt.Errorf("managedkafka.NewClient got err: %w", err)
	}
	defer client.Close()

	locationPath := fmt.Sprintf("projects/%s/locations/%s", projectID, region)
	req := &managedkafkapb.ListClustersRequest{
		Parent: locationPath,
	}
	clusterIter := client.ListClusters(ctx, req)
	for {
		res, err := clusterIter.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return fmt.Errorf("clusterIter.Next() got err: %w", err)
		}
		fmt.Fprintf(w, "Got cluster: %v", res)
	}
	return nil
}

Java

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.managedkafka.v1.Cluster;
import com.google.cloud.managedkafka.v1.LocationName;
import com.google.cloud.managedkafka.v1.ManagedKafkaClient;
import java.io.IOException;

public class ListClusters {

  public static void main(String[] args) throws Exception {
    // TODO(developer): Replace these variables before running the example.
    String projectId = "my-project-id";
    String region = "my-region"; // e.g. us-east1
    listClusters(projectId, region);
  }

  public static void listClusters(String projectId, String region) throws Exception {
    try (ManagedKafkaClient managedKafkaClient = ManagedKafkaClient.create()) {
      LocationName locationName = LocationName.of(projectId, region);
      // This operation is being handled synchronously.
      for (Cluster cluster : managedKafkaClient.listClusters(locationName).iterateAll()) {
        System.out.println(cluster.getAllFields());
      }
    } catch (IOException | ApiException e) {
      System.err.printf("managedKafkaClient.listClusters got err: %s", e.getMessage());
    }
  }
}

What's next?

Apache Kafka® is a registered trademark of The Apache Software Foundation or its affiliates in the United States and/or other countries.