Dataproc Service Account Based Secure Multi-tenancy

Dataproc Service Account based Secure Multi-tenancy (called "secure multi-tenancy", below) enables you to share a cluster with multiple users, with a set of users mapped to service accounts when the cluster is created. With secure multi-tenancy, users can submit interactive workloads to the cluster with isolated user identities.

When a user submits a job to the cluster, the job:

  • runs as a specific OS user with a specific Kerberos principal

  • accesses Google Cloud resources using the mapped service account credentials

Considerations and Limitations

When you create a cluster with secure multi-tenancy enabled:

  • You can submit jobs only through the Dataproc Jobs API.

  • The cluster is available only to users with mapped service accounts. For example, unmapped users cannot run jobs on the cluster.

  • Service accounts can be mapped only to Google users, not Google groups.

  • The Dataproc Component Gateway is not enabled.

  • Direct SSH access to the cluster and Compute Engine features, such as the ability to run startup scripts on cluster VMs, are blocked. Also, jobs cannot run with sudo privileges.

  • Kerberos is enabled and configured on the cluster for secure intra-cluster communication. End user authentication through Kerberos is not supported.

  • Dataproc Workflows are not supported.

Creating a secure multi-tenancy cluster

To create a Dataproc secure multi-tenancy cluster, use the --secure-multi-tenancy-user-mapping flag to specify a list of user-to-service-account mappings.


The following command creates a cluster, with user mapped to service account and user mapped to service account

gcloud dataproc clusters create my-cluster \
    --secure-multi-tenancy-user-mapping="," \
    --scopes= \ \
    --region=region \
    other args ...

Alternatively, you can store the list of user-to-service-account mappings in a local or Cloud Storage YAML or JSON file. Use the --identity-config-file flag to specify the file location.

Sample identity config file:


Sample command to create the cluster using the --identity-config-file flag:

gcloud dataproc clusters create my-cluster \
    --identity-config-file=local or "gs://bucket" /path/to/identity-config-file \
    --scopes= \ \
    --region=region \
    other args ...


  • As shown in the above commands, cluster --scopes must include at least, which is necessary for the cluster service account to perform impersonation.

  • The cluster service account must have permissions to impersonate the service accounts mapped to the users (see Service account permissions).

  • Recommendation: Use different cluster service accounts for different clusters to allow each cluster service account to impersonate only a limited, intended group of mapped user service accounts.