Troubleshoot managed workload identity authentication for GKE

This document describes resolutions for common managed workload identities errors.

You can check the status of a Pod by running the following command:

kubectl describe pod

Pod fails to be deployed with certificates

If the Google Kubernetes Engine (GKE) workload Pod has failed to be deployed with the mounted certificates, use the following command to check the Pod status:

kubectl describe pod POD_NAME -n POD_NAMESPACE

Permissions-related errors

This section describes errors related to incorrect permissions.

Permission denied while issuing the certificate: failed to issue the certificate from the GKE Auth

This error looks similar to the following:

Permission denied while issuing the certificate: failed to issue the certificate from the GKE Auth: rpc error: code = PermissionDenied desc = Permission 'privateca.certificates.createForSelf' denied on 'projects/PROJECT_NUMBER/locations/REGION/caPools/ CA_POOL_ID'

The error occurred because the workload identity pool is missing the CA Service Workload Certificate Requester role (roles/privateca.workloadCertificateRequester) on the subordinate CA pool.

To resolve this error, grant the role to the pool:

gcloud privateca pools add-iam-policy-binding SUBORDINATE_CA_POOL_ID \
   --project=CA_POOL_PROJECT_ID \
   --location=REGION \
   --role=roles/privateca.workloadCertificateRequester \
   --member="principal://iam.googleapis.com/projects/CA_POOL_PROJECT_NUMBER/name/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog"

Replace the following:

  • SUBORDINATE_CA_POOL_ID: the subordinate CA pool ID
  • CA_POOL_PROJECT_ID: the project ID of the root CA pool
  • REGION: the subordinate CA region
  • CA_POOL_PROJECT_NUMBER: the CA pool project number
  • PROJECT_ID: the workload identity pool project ID

MountVolume.SetUp failed for volume

This error looks similar to the following:

MountVolume.SetUp failed for volume "fleet-spiffe-credentials" : rpc error: code = Internal desc = unable to mount volume: while creating volume: while loading trust bundles: rpc error: code = Internal desc = while getting trust map from the trustBundleWatcher: rpc error: code = Internal desc = no GKEClusterTrustBundle objects found with fleet WIP nickname fleet-project/svc.id.goog

This error occurred because the workload identity pool is missing the Private CA Pool Reader role (roles/privateca.poolReader) on the subordinate CA pool.

To resolve this error, grant the role to the pool and wait a few minutes for the trust bundles to be loaded:

gcloud privateca pools add-iam-policy-binding SUBORDINATE_CA_POOL_ID \
 --location=REGION \
 --role=roles/privateca.poolReader \
 --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/name/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog"

Replace the following:

  • SUBORDINATE_CA_POOL_ID: the subordinate CA pool ID
  • REGION: the subordinate CA region
  • PROJECT_NUMBER: the workload identity pool project number
  • PROJECT_ID: the workload identity pool project ID

You can use the following command to check the existence of the trust bundles:

kubectl get gkeclustertrustbundle

When the trust bundles are loaded, the response looks like the following:

NAME                                         AGE
svc.id.goog-PROJECT_ID.svc.id.goog   5m35s

In the output, PROJECT_ID is the project ID of the fleet that your trust root is associated with.

Certificate errors

The errors in this section are related to certificate configuration.

Several errors can occur if managed workload identity certificates aren't generated. To check for the existence of these certificates on Pod-mounted volumes, do the following:

  1. Connect to the pod.

    # Open the pod sh interface
kubectl exec -it $pod_name -n $pod_namespace -- sh

  2. List and inspect the credential files.

    # Look up the volume contents:
ls -la /var/run/secrets/workload-spiffe-credentials
cat /var/run/secrets/workload-spiffe-credentials/certificates.pem
cat /var/run/secrets/workload-spiffe-credentials/ca_certificates.pem
cat /var/run/secrets/workload-spiffe-credentials/private_key.pem
cat /var/run/secrets/workload-spiffe-credentials/trust_bundles.json

You can also inspect a Pod's certificate provision status by inspecting k8s_container logs for the `gke-metadata-server container through the Google Cloud console.

Failed precondition while issuing the certificate

The Pod description can include the following message:

Failed precondition while issuing the certificate: failed to issue the certificate from the GKE Auth: rpc error: code = FailedPrecondition

This error occurred because the workload identity pool wasn't configured with a certificate issuance configuration (CIC).

To resolve this error, create a CIC and use it to update the workload identity pool.