Access control with IAM and Kafka ACLs

This document describes the two main layers for access control options for Google Cloud Managed Service for Apache Kafka:

• Google Cloud Identity and Access Management (IAM).

• Standard Apache Kafka access control lists (ACLs).

IAM

IAM controls access at the Google Cloud resource level:

• IAM controls determine who can manage your Managed Service for Apache Kafka resources like clusters, topics, or ACLs using Google Cloud APIs and tools. Examples of such tools are the Google Cloud console, gcloud CLI, or client libraries.

• These controls also determine who has permission to initially connect to your Managed Service for Apache Kafka cluster when using standard Apache Kafka clients.

For more information about IAM, see the IAM documentation.

Apache Kafka ACLs

Apache Kafka ACLs provide fine-grained access control within the Kafka cluster. Managed Service for Apache Kafka enables the out-of-the-box StandardAuthorizer, which stores ACLs in the KRaft-based Kafka cluster metadata.

• These ACLs control which authenticated users can perform specific operations on specific Kafka resources, like producing or consuming messages on a topic.

• These ACLs are useful to control interactions with the cluster using standard Apache Kafka clients, which are only subject to a cluster-level IAM check for the initial connection.

How access control works

Authorization for access using Google Cloud APIs is entirely handled by IAM and default system ACLs in the cluster. Authorization for access from open source Apache Kafka clients works in the following way:

1. IAM check. Before any Apache Kafka ACLs are evaluated, IAM first checks if the connecting principal has the necessary permission, like managedkafka.clusters.connect, to connect to the cluster. If the IAM check fails, the connection is denied.

2. Kafka ACL check. If the IAM check passes, the client interacts with Kafka's standard ACL authorizer within the cluster. The authorizer evaluates the relevant Apache Kafka ACLs to authorize specific operations requested by the principal, such as producing to a topic or consuming from a group.

For comprehensive security, you need to configure the following:

• IAM permissions for connection and management access.

• Kafka ACLs for in-cluster data access and operations from open-source Apache Kafka clients.

For example, assume that you want to prevent a principal from editing topics. You can achieve this in two ways:

• Fully through IAM. Deny the principal both the Managed Kafka Topic Editor (roles/managedkafka.topicEditor) and Managed Kafka Client (roles/managedkafka.client) roles. This configuration completely restricts topic editing through Google Cloud APIs and prevents any Kafka API access.

• Using Kafka ACLs in conjunction with IAM. If the principal requires the Managed Kafka Client (roles/managedkafka.client) role for actions like publishing to a topic, but you want to restrict them from editing topics, you must use Kafka ACLs. Specifically, restrict the following operations:

  • Create (for topic creation) at the cluster level.

  • Alter, AlterConfigs, Delete (for topic modification and deletion) at the topic level.

You can choose the appropriate method based on whether the principal needs access to the Kafka APIs.

Usage of IAM

Similar to other Google Cloud products, Managed Service for Apache Kafka supports three types of roles:

• Basic roles: highly permissive roles that existed prior to the introduction of IAM. For more information about basic roles, see Basic roles.

• Predefined roles: give granular access to specific Google Cloud resources. For more information about predefined roles, see Predefined roles. The Managed Service for Apache Kafka predefined roles are included in a later part of this section.

• Custom roles: help you enforce the principle of least privilege. For more information about custom roles, see Custom roles.

In Managed Service for Apache Kafka, you can configure access control with IAM conditions at the level of the cluster, consumer group, or topic.

Every Managed Service for Apache Kafka method requires a corresponding permission. For a list of the permissions and roles that IAM supports, see the section Predefined roles.

Set access control at the project level

To set access controls at the project level, see Manage access to projects, folders, and organizations.

Set access control at the resource level

Some Managed Service for Apache Kafka requests, such as cluster creation or updates, are long-running operations. To allow a principal to perform these actions, grant them access to managedkafka.googleapis.com/Operation resources in addition to the specific cluster resource.

This configuration ensures the principal can both initiate the operation and monitor its progress.

The following is an example of an IAM condition set on a topic named "test-topic":

{'expression': 'resource.name.endsWith('test-topic') 'title': 'SampleIAMCondition'}`.

This example shows an IAM condition that checks if a resource name ends with test-topic. If it does, the condition is true. This specific condition is titled SampleIAMCondition and could be used in an IAM policy to restrict access to only this particular topic.

Certain Managed Service for Apache Kafka requests, such as cluster creation or updates, return long-running operations. To grant access at the cluster level, include access for all resources of type managedkafka.googleapis.com/Operation in addition to the per-cluster resource condition. This process ensures the principal can initiate the operation and monitor its progress.

Usage of Apache Kafka ACLs

Apache Kafka ACL bindings have the following format:

Principal P is [Allowed/Denied] Operation O From Host H on any resource matching Resource Pattern RP.

The following is a list of important information about the format:

• Principal(P): the user or service account requesting access.

• Permission type(Allowed/Denied): whether the ACL binding allows or denies access. Deny bindings take precedence.

• Operation(O): the action performed such as Read, Write, or Create. For information about which operations apply to which resources for various Kafka protocols, see [the Apache Kafka documentation](https://kafka.apache.org/documentation/#operations_resources_and_protocols).

• Host(H): the machine from which the request originates. Since Managed Service for Apache Kafka translates client network addresses, use of hosts other than '*' is not supported.

• Resource Pattern(RP): a pattern used to match specific resources. A resource pattern consists of a resource type, a resource name, and a pattern type (LITERAL or PREFIXED).

Default access

Managed Service for Apache Kafka clusters operate with the Apache Kafka property allow.everyone.if.no.acl.found set to true. The presence or absence of Kafka ACLs directly dictates the level of access to resources:

• If no Kafka ACLs are defined for a specific resource like a topic, all authenticated principals are granted access to it. This configuration lets Managed Service for Apache Kafka clusters be immediately operable, without the requirement to configure ACLs.

• As soon as you define any Kafka ACL for that resource, access becomes restricted. Only principals explicitly granted permission using an ALLOW entry in a Kafka ACL can access matching resources (unless specifically blocked by a DENY entry).

Managed Service for Apache Kafka gives its service agent administrative access to your cluster. This access allows the service agent to perform operations requested by the Managed Service for Apache Kafka API, regardless of other ACLs configured in the cluster. Managed Service for Apache Kafka accomplishes this by modifying the StandardAuthorizer implementation. The modification grants the service agent permissions similar to super users, except for read and write operations. You cannot change this configuration.

Kafka principals

Kafka principals for Managed Service for Apache Kafka clusters are specified as Google Cloud accounts, with the Kafka StandardAuthorizer prefix "User:". For example, to grant access to the service account test-kafka-client@test-project.iam.gserviceaccount.com, use the Kafka principal "User:test-kafka-client@test-project.iam.gserviceaccount.com".

Kafka ACL principals must specify a user, a service account, or an individual IAM principal, but not a group or principal set. Kafka ACLs don't support resolving group memberships for Google Cloud principals. For example, authorizing a service account as a member of an allowed Google Group is not supported.

To create an ACL that applies to all members of a Google Group or principal set, you can use a proxy service account principal and service account impersonation:

1. Create a service account to use as a proxy for the group.

2. Grant the Google Group or principal set the Service Account Token Creator role on the service account. See Manage access to service accounts

3. Add Kafka ACLs for the proxy service account. An example principal would be: User:group-proxy@test-project.iam.gserviceaccount.com.

4. Use service account impersonation in Kafka clients to authenticate to Kafka as the service account. Google Cloud IAM authorizes an individual principal as a member of the group that's allowed to impersonate the proxy service account. And Kafka authorizes the proxy service account against existing ACLs in the cluster.

Kafka operations for producers and consumers

An operation is an action performed on a resource. For each resource, an operation is mapped to one or more Kafka protocol requests for that resource. For example, a READ operation for the topic resource type is mapped to Fetch, OffsetCommit, and TxnOffsetCommit Apache Kafka protocols.

To grant a principal producer access on a topic, perform the following steps:

1. On the topic resource, allow the WRITE and CREATE operations.

2. If using transactional IDs, on the transactional ID resource, allow the WRITE operation.

To grant a principal consumer access on a topic, perform the following steps:

1. On the topic resource, allow the READ operation.

2. On the consumer group resource, allow the READ operation.

For more information about the valid operations on resources that the Kafka API supports, see Operations, resources, and protocols.

Configure ACLS for default-deny behavior

Managed Kafka clusters are configured with allow.everyone.if.no.acl.found = true. So by default, if no ACLs are set on a resource, all principals can access the resource.

To configure a default-deny behavior similar to that in IAM, you can first configure access for an administrator user on all resources in the cluster. As a result, every resource gets an ACL defined on it, and the allow.everyone.if.no.acl.found behavior is suppressed. By default, any principal not explicitly allowed by an ALLOW ACL is denied access.

For example, to set ACLs on all resources in the cluster for the service account clusterAdmin@test-project.iam.gserviceaccount.com, create the following ACL entries.

The following gcloud CLI command grants a service account named clusterAdmin@test-project.iam.gserviceaccount.com full administrative access (--operation=ALL) to a specific Kafka cluster located in a particular region. This permission allows the service account to perform any operation on the cluster from any host.

gcloud managed-kafka acls add-acl-entry cluster /
    --principal=`User:clusterAdmin@test-project.iam.gserviceaccount.com` /
    --operation=ALL /
    --permission-type=ALLOW /
    --host=* /
    --cluster=CLUSTER_ID /
    --location=LOCATION

The following gcloud CLI command grants a service account named clusterAdmin@test-project.iam.gserviceaccount.com full administrative access (--operation=ALL) to all topics within a specific Kafka cluster located in a particular region. This permission allows the service account to perform any operation on all topics from any host.

gcloud managed-kafka acls add-acl-entry allTopics /
    --principal=`User:clusterAdmin@test-project.iam.gserviceaccount.com` /
    --operation=ALL /
    --permission-type=ALLOW /
    --host=* /
    --cluster=CLUSTER_ID /
    --location=LOCATION

The following gcloud CLI command grants a service account named clusterAdmin@test-project.iam.gserviceaccount.com full administrative access (--operation=ALL) to all consumer groups within a specific Kafka cluster located in a particular region. This permission allows the service account to perform any operation on all consumer groups from any host.

gcloud managed-kafka acls add-acl-entry allConsumerGroups /
    --principal=`User:clusterAdmin@test-project.iam.gserviceaccount.com` /
    --operation=ALL /
    --permission-type=ALLOW /
    --host=* /
    --cluster=CLUSTER_ID /
    --location=LOCATION

The following gcloud CLI command grants a service account named clusterAdmin@test-project.iam.gserviceaccount.com full administrative access (--operation=ALL) to all transactional IDs within a specific Kafka cluster located in a particular region. This permission allows the service account to perform any operation on all transactional IDs from any host.

gcloud managed-kafka acls add-acl-entry allTransactionalIds /
    --principal=`User:clusterAdmin@test-project.iam.gserviceaccount.com` /
    --operation=ALL /
    --permission-type=ALLOW /
    --host=* /
    --cluster=CLUSTER_ID /
    --location=LOCATION

The following is a list of important information about the commands:

• --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com': the principal to whom the ACL applies. The principal is a Google Cloud account, with the Kafka StandardAuthorizer prefix User:.

• --operation=all: the Kafka operation being granted, which is full access in this case.

• --permission_type=ALLOW: this ACL entry grants access.

• --host='*': the host from which the principal can access the resource. '*' grants access from any host. Managed Service for Apache Kafka only supports ACLs with host '*'.

• CLUSTER_ID: the name of your Managed Service for Apache Kafka cluster.

• LOCATION: the Google Cloud region where your Managed Service for Apache Kafka cluster is located such as us-central1.

Configure ACLs

You can configure Apache Kafka ACLs with the Managed Service for Apache Kafka ACL APIs, or with open-source Apache Kafka tools such as Apache Kafka authorizer cli kafka-acls.sh, or Admin Client.

Managed Service for Apache Kafka organizes ACLs by Kafka resource patterns. The resource pattern is defined by the following:

• Resource type: cluster, topic, consumer group, or transactional ID

• Pattern type: literal, or prefixed (all resources whose name start with the provided string)

• Resource name: the resource name or prefix to which the ACL entries apply.

A Managed Service for Apache Kafka ACL resource represents all access control configured for a single Kafka resource pattern, as a repeated list of ACL entries. The name of the ACL resource uniquely identifies the resource pattern of the ACL binding. For more information, see ACL ID.

You can manage Managed Service for Apache Kafka ACLs at the ACL resource level (all ACL entries for a resource pattern), or incrementally by adding and removing individual ACL entries for an ACL resource pattern.

For more information, see Create a Managed Kafka ACL and Add a Managed Kafka ACL entry.

Allow reading from a topic

To allow open-source Apache Kafka clients running as the service account test-kafka-client@test-project.iam.gserviceaccount.com to read from the topic topic-name, create a Managed Service for Apache Kafka ACL entry using the managed-kafka acls add-acl-entry command:

gcloud managed-kafka acls add-acl-entry topic/topic-name \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=READ \
    --permission_type=ALLOW \
    --host='*'

The following is a list of important information about the command:

• topic/topic-name: specifies the Managed Service for Apache Kafka topic to which you want to grant access. Replace topic-name with the actual name of your topic. The topic/ prefix indicates that this ACL entry applies to a specific (literal) topic resource pattern.

• LOCATION: the Google Cloud region where your Managed Service for Apache Kafka cluster is located, such as us-central1.

• CLUSTER_ID: the name of your Managed Service for Apache Kafka cluster.

• --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com': the principal to whom the ACL applies. The principal is a Google Cloud account, with the Kafka StandardAuthorizer prefix User:.

• --operation=READ: the Kafka operation being granted, which is READ in this case.

• --permission_type=ALLOW: indicates that this ACL entry grants access.

• --host='*': specifies the host from which the principal can access the resource. '*' grants access from any host. Managed Service for Apache Kafka only supports ACLs with host '*'.

To remove read access, you can use the remove-acl-entry command with the same parameters.

gcloud managed-kafka acls remove-acl-entry topic/topic-name \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=READ \
    --permission_type=ALLOW \
    --host='*'

Allow writing to all topics with a common prefix

To allow open-source Apache Kafka clients running as the service account test-kafka-client@test-project.iam.gserviceaccount.com to write to all topics whose name starts with the prefix topic-prefix, add a Managed Kafka ACL entry as follows:

gcloud managed-kafka acls add-acl-entry topicPrefixed/topic-prefix \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=WRITE \
    --permission_type=ALLOW \
    --host='*'

The following is a list of important information about the command:

• topicPrefixed/topic-prefix: specifies the Managed Service for Apache Kafka resource pattern to which you want to grant access. Replace topic-prefix with the actual prefix of your topics. The topicPrefixed/ prefix indicates that this ACL entry applies to a prefixed resource pattern: all topics matching the given prefix.

• PROJECT: the ID of your Google Cloud project where your Managed Service for Apache Kafka cluster is located.

• LOCATION: the Google Cloud region where your Managed Service for Apache Kafka cluster is located such as us-central1.

• CLUSTER_ID: the name of your Managed Service for Apache Kafka cluster.

• --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com': the principal to whom the ACL applies. The principal is a Google Cloud account, with the Kafka StandardAuthorizer prefix User:.

• --operation=WRITE: the Kafka operation being granted, which is WRITE in this case.

• --permission_type=ALLOW: this ACL entry grants access.

• --host='*': the host from which the principal can access the resource. '*' grants access from any host. Managed Service for Apache Kafka only supports ACLs with host '*'.

To remove write access for this service account, remove the ACL entry:

gcloud managed-kafka acls remove-acl-entry topicPrefixed/topic-prefix \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=WRITE \
    --permission_type=ALLOW \
    --host='*'

Deny modifying all topics

To prevent open-source Apache Kafka clients running as the service account test-kafka-client@test-project.iam.gserviceaccount.com from modifying all topics in a cluster, you can create a Managed Service for Apache Kafka ACL resource with a list of AclEntry resources to deny the ALTER, ALTER_CONFIGS, and DELETE operations on all topics. This approach defines the required state in a single configuration.

Alternatively, you can achieve the same result by imperatively adding three separate ACL entries using the gcloud managed-kafka acls add-acl-entry command. This method involves running the command to deny access to each of the operations ALTER, ALTER_CONFIGS, and DELETE, as follows:

gcloud managed-kafka acls add-acl-entry allTopics \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=ALTER \
    --permission_type=DENY \
    --host='*'

gcloud managed-kafka acls add-acl-entry allTopics \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=ALTER_CONFIGS \
    --permission_type=DENY \
    --host='*'

gcloud managed-kafka acls add-acl-entry allTopics \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=DELETE \
    --permission_type=DENY \
    --host='*'

The following information applies to each of the add-acl-entry commands:

• allTopics: specifies that this ACL applies to all topics within the Managed Service for Apache Kafka cluster.

• LOCATION: the Google Cloud region where your Managed Service for Apache Kafka cluster is located such as us-central1.

• CLUSTER_ID: the name of your Managed Service for Apache Kafka cluster.

• --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com': specifies the principal to whom the ACL applies. The principal is a Google Cloud account, with the Kafka StandardAuthorizer prefix User:.

• --operation: specifies the Kafka operation being denied:

  • ALTER: includes actions like changing the number of partitions or replication factor.

  • ALTER_CONFIGS: includes modifying topic-level configurations.

  • DELETE: includes deleting topics.

• --permission_type=DENY: indicates that these ACL entries block access for the specified operations.

• --host='*': specifies that this denial applies regardless of the host from which the request originates. Managed Service for Apache Kafka only supports ACLs with host '*'.

To remove these restrictions, use the remove-acl-entry command for each of the added entries, using the same parameters. For example, to re-allow topic deletion:

gcloud managed-kafka acls remove-acl-entry allTopics \
    --cluster=CLUSTER_ID /
    --location=LOCATION
    --principal='User:test-kafka-client@test-project.iam.gserviceaccount.com' \
    --operation=DELETE \
    --permission_type=DENY \
    --host='*'

Troubleshooting ACLs

The Apache Kafka Standard Authorizer writes audit logs by default on authorization denials. If you receive a Kafka Authorization error, you can confirm the principal, resource, and operation that was denied by searching the cluster logs for StandardAuthorizerData logAuditMessage.

For example, here is a sample cluster log.

org.apache.kafka.metadata.authorizer.StandardAuthorizerData logAuditMessage\n
INFO: Principal = User:556291496362-compute@developer.iam.gserviceaccount.com is
Denied operation = DESCRIBE from host = 172.16.0.20 on resource = Topic:LITERAL:t1
for request = Metadata with resourceRefCount = 1 based on rule DefaultDeny

Managed Service for Apache Kafka predefined roles

The following table lists the Managed Service for Apache Kafka predefined roles.

Permissions associated with Managed Kafka APIs

The following tables outline the required permissions for each Managed Kafka API method related to cluster, topics, and consumer groups. It shows which IAM permission is needed to call each method, along with a description of what the method does.

Permissions for clusters

Permissions for topics

Permissions for consumer groups

Permissions for Connect clusters

Permissions for connectors

Permissions for ACLs

What's next

• IAM documentation.

• Granting, changing, and revoking access to resources.

• Apache Kafka documentation on ACLs.