You can update a Google Cloud Managed Service for Apache Kafka consumer group to modify the offsets for a list of topic partitions. This lets you control which messages the consumers in the group receive.
To update a consumer group, you can use the Google Cloud CLI, the client library, the Managed Kafka API, or the open source Apache Kafka APIs. The Google Cloud console is not supported for editing a consumer group.
Before you begin
To update a consumer group, first ensure it is not actively consuming messages. A consumer group is automatically deleted by Kafka if it has never consumed any messages, or when the last committed offset has expired after offsets.retention.minutes.
Follow these steps before you update a consumer group:
Send some messages to the topic from which your consumer group is reading messages.
Start your consumer group to process a few messages.
Stop all your consumers from consuming messages. To stop a consumer, press Control+C.
For more information about sending and consuming messages, see Use the Kafka command line tools.
Required roles and permissions to update a consumer group
To get the permissions that you need to edit your consumer groups,
ask your administrator to grant you the
Managed Kafka Consumer Group Editor (roles/managedkafka.consumerGroupEditor
) IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to edit your consumer groups. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to edit your consumer groups:
-
Update consumer groups:
managedkafka.consumerGroups.update
You might also be able to get these permissions with custom roles or other predefined roles.
For more information about the Managed Kafka Consumer Group Editor role, see Managed Service for Apache Kafka predefined roles.
Update a consumer group
Ensure you have completed the steps in the Before you begin section.
To update a consumer group, follow these steps:
gcloud
-
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Run the
gcloud managed-kafka consumer-groups update
command:gcloud managed-kafka consumer-groups update CONSUMER_GROUP_ID \ --cluster=CLUSTER_ID \ --location=LOCATION \ --topics-file=TOPICS_FILE
Replace the following:
-
CLUSTER_ID: The ID or name of the cluster.
-
LOCATION: The location of the cluster.
-
CONSUMER_GROUP_ID: The ID or name of the consumer group.
-
TOPICS_FILE: This setting specifies the location of the file containing the configuration of topics to be updated for the consumer group. The file can be in JSON or YAML format. It can be a file path or directly include the JSON or YAML content.
The topic file uses a JSON structure to represent a
ConsumerGroup
topics map, in the form{ topicName1: {ConsumerPartitionMetadata}, topicName2:{ConsumerPartitionMetadata}}
. For each topic,ConsumerPartitionMetadata
provides the offset and metadata for each partition.To set the offset for a single partition (partition 0) in a topic named
topic1
to 10, the JSON configuration would look like:{"topic1": {"partitions": { 0 : { "offset": 10, "metadata": ""}}}}
The following is an example of the contents of a
topics.json
file:{ "projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME/TOPICS/TOPIC_NAME": { "partitions": { "1": { "offset": "1", "metadata": "metadata" }, "2": { "offset": "1", "metadata": "metadata" } } }, "projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME/TOPICS/OTHER_TOPIC_NAME": { "partitions": { "1": { "offset": "1", "metadata": "metadata" } } } }
-
TOPIC_PATH: When specifying topics in JSON or YAML file, include the full topic path which can be obtained from running the
gcloud managed-kafak topics describe
command and of the formatprojects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_ID/topics/topic
. .