Set up service security with proxyless gRPC

This guide shows you how to configure a security service for proxyless gRPC service mesh.

Requirements

Before you configure service security for the gRPC proxyless service mesh, make sure that you meet the following requirements.

• Your deployment meets the requirements in Prepare to set up on service routing APIs with Envoy and proxyless workloads.
• You must use xDS v3.
• You have access to the required xDS version and certificate provider functionality with one of the following languages:
  • gRPC Java
  • gRPC C++
  • gRPC Python
  • gRPC Go You can find the required language versions on github
• You have access to the bootstrap generator, version 0.16.0. The bootstrap generator image is located in the Google Cloud container repository.
• You meet all of the prerequisites for gRPC proxyless service mesh load balancing.
• You have sufficient permissions to create or update the Cloud Service Mesh and Google Cloud service mesh resources to use PSM security. For complete information about the required permissions, see Prepare to set up Cloud Service Mesh with proxyless gRPC services.
• You have the required permissions to use Certificate Authority Service, which are described in Create certificate authorities to issue certificates

Configure Identity and Access Management

You must have the required permissions to use Google Kubernetes Engine. At a minimum, you must have the following roles:

• roles/container.clusterAdmin GKE role
• roles/compute.instanceAdmin Compute Engine role
• roles/iam.serviceAccountUser role

To create the resources required for the setup, you must have the compute.NetworkAdmin role. This role contains all the necessary permissions to create, update, delete, list, and use (that is, referencing this in other resources) the required resources. If you are the owner-editor of your project, you automatically have this role.

Note that the networksecurity.googleapis.com.clientTlsPolicies.use and networksecurity.googleapis.com.serverTlsPolicies.use are not enforced when you reference these resources in the backend service resource.

If this is enforced in the future and you are using the compute.NetworkAdmin role, then you won't notice any issues when this check is enforced.

If you are using custom roles and this check is enforced in the future, you must make sure to include the respective .use permission. Otherwise, in the future, you might find that your custom role does not have the necessary permissions to refer to clientTlsPolicy or serverTlsPolicy from the backend service.

Prepare for setup

Proxyless service mesh (PSM) security adds security to a service mesh that is set up for load balancing per the proxyless gRPC services documentation. In a proxyless service mesh, a gRPC client uses the scheme xds: in the URI to access the service, which enables the PSM load balancing and endpoint discovery features.

Update gRPC clients and servers to the correct version

Build or rebuild your applications using the minimum supported gRPC version for your language.

Update the bootstrap file

gRPC applications use a single bootstrap file, which must have all of the fields that are required by gRPC client- and server-side code. A bootstrap generator automatically generates the bootstrap file to include flags and values that PSM security needs. For more information, see the Bootstrap file section, which includes a sample bootstrap file.

Setup overview

This setup process is an extension of the Cloud Service Mesh setup with GKE and proxyless gRPC services. Existing unmodified steps of that setup procedure are referenced wherever they apply.

The main enhancements to the Cloud Service Mesh setup with GKE are as follows:

1. Setting up CA Service, in which you create private CA pools and the required certificate authorities.
2. Creating a GKE cluster with GKE Workload Identity Federation for GKE and mesh certificates features and CA Service integration.
3. Configuring mesh certificate issuance on the cluster.
4. Creating the client and server service accounts.
5. Setting up the example server that uses xDS APIs and xDS server credentials to acquire security configuration from Cloud Service Mesh.
6. Setting up the example client that uses xDS credentials.
7. Updating the Cloud Service Mesh configuration to include security configuration.

You can see code examples for using xDS credentials at the following locations:

• Java.
• C++
• Go
• Python

Update the Google Cloud CLI

To update the Google Cloud CLI, run the following command:

gcloud components update

Set up environment variables

In this guide, you use Cloud Shell commands, and repeating information in the commands is represented by various environment variables. Set your specific values to the following environment variables in the shell environment before you execute the commands. Each comment line indicates the meaning of the associated environment variable.

# Your project ID
PROJECT_ID=PROJECT_ID

# GKE cluster name and zone for this example.
CLUSTER_NAME=CLUSTER_NAME
ZONE=ZONE
gcloud config set compute/zone $ZONE

# GKE cluster URL derived from the above
GKE_CLUSTER_URL="https://container.googleapis.com/v1/projects/${PROJECT_ID}/locations/${ZONE}/clusters/${CLUSTER_NAME}"

# Workload pool to be used with the GKE cluster
WORKLOAD_POOL="${PROJECT_ID}.svc.id.goog"

# Kubernetes namespace to run client and server demo.
K8S_NAMESPACE='default'
DEMO_BACKEND_SERVICE_NAME='grpc-gke-helloworld-service'

# Compute other values
# Project number for your project
PROJNUM=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")

# VERSION is the GKE cluster version. Install and use the most recent version
# from the rapid release channel and substitute its version for
# CLUSTER_VERSION, for example:
# VERSION=latest available version
# Note that the minimum required cluster version is 1.21.4-gke.1801.
VERSION="CLUSTER_VERSION"
SA_GKE=service-${PROJNUM}@container-engine-robot.iam.gserviceaccount.com

Enable access to required APIs

This section tells you how to enable access to the necessary APIs.

1. Run the following command to enable the Cloud Service Mesh and other APIs required for proxyless gRPC service mesh security.

   gcloud services enable \
       container.googleapis.com \
       cloudresourcemanager.googleapis.com \
       compute.googleapis.com \
       trafficdirector.googleapis.com \
       networkservices.googleapis.com \
       networksecurity.googleapis.com \
       privateca.googleapis.com \
       gkehub.googleapis.com
   

2. Run the following command to allow the default service account to access the Cloud Service Mesh security API.

   GSA_EMAIL=$(gcloud iam service-accounts list --format='value(email)' \
       --filter='displayName:Compute Engine default service account')
   
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
     --member serviceAccount:${GSA_EMAIL} \
      --role roles/trafficdirector.client
   

Create or update a GKE cluster

Cloud Service Mesh service security depends on the CA Service integration with GKE. The GKE cluster must meet the following requirements in addition to the requirements for setup:

• Use a minimum cluster version of 1.21.4-gke.1801. If you need features that are in a later version, you can obtain that version from the rapid release channel.
• The GKE cluster must be enabled and configured with mesh certificates, as described in Creating certificate authorities to issue certificates.

1. Create a new cluster that uses Workload Identity Federation for GKE. If you are updating an existing cluster, skip to the next step. The value you give for --tags must match the name passed to the --target-tags flag for the firewall-rules create command in the section Configuring Cloud Service Mesh with Cloud Load Balancing components.

   # Create a GKE cluster with GKE managed mesh certificates.
   gcloud container clusters create CLUSTER_NAME \
     --release-channel=rapid \
     --scopes=cloud-platform \
     --image-type=cos_containerd \
     --machine-type=e2-standard-2 \
     --zone=ZONE \
     --workload-pool=PROJECT_ID.svc.id.goog \
     --enable-mesh-certificates \
     --cluster-version=CLUSTER_VERSION \
     --enable-ip-alias \
     --tags=allow-health-checks \
     --workload-metadata=GKE_METADATA
   

   Cluster creation might take several minutes to complete.

2. If you are using an existing cluster, turn on Workload Identity Federation for GKE and GKE mesh certificates. Make sure that the cluster was created with the --enable-ip-alias flag, which cannot be used with the update command.

   gcloud container clusters update CLUSTER_NAME \
     --enable-mesh-certificates
   

3. Run the following command to switch to the new cluster as the default cluster for your kubectl commands:

   gcloud container clusters get-credentials CLUSTER_NAME \
     --zone ZONE
   

Register clusters with a fleet

Register the cluster that you created or updated in Creating a GKE cluster with a fleet. Registering the cluster makes it easier for you to configure clusters across multiple projects.

Note that these steps can take up to ten minutes each to complete.

1. Register your cluster with the fleet:

   gcloud container fleet memberships register CLUSTER_NAME \
     --gke-cluster=ZONE/CLUSTER_NAME \
     --enable-workload-identity --install-connect-agent \
     --manifest-output-file=MANIFEST-FILE_NAME
   

   Replace the variables as follows:

   • CLUSTER_NAME: Your cluster's name.
   • ZONE: Your cluster's zone.
   • MANIFEST-FILE_NAME: The path where these commands generate the manifest for registration.

   When the registration process succeeds, you see a message such as the following:

   Finished registering the cluster CLUSTER_NAME with the fleet.

2. Apply the generated manifest file to your cluster:

   kubectl apply -f MANIFEST-FILE_NAME
   

   When the application process succeeds, you see messages such as the following:

   namespace/gke-connect created
   serviceaccount/connect-agent-sa created
   podsecuritypolicy.policy/gkeconnect-psp created
   role.rbac.authorization.k8s.io/gkeconnect-psp:role created
   rolebinding.rbac.authorization.k8s.io/gkeconnect-psp:rolebinding created
   role.rbac.authorization.k8s.io/agent-updater created
   rolebinding.rbac.authorization.k8s.io/agent-updater created
   role.rbac.authorization.k8s.io/gke-connect-agent-20210416-01-00 created
   clusterrole.rbac.authorization.k8s.io/gke-connect-impersonation-20210416-01-00 created
   clusterrolebinding.rbac.authorization.k8s.io/gke-connect-impersonation-20210416-01-00 created
   clusterrolebinding.rbac.authorization.k8s.io/gke-connect-feature-authorizer-20210416-01-00 created
   rolebinding.rbac.authorization.k8s.io/gke-connect-agent-20210416-01-00 created
   role.rbac.authorization.k8s.io/gke-connect-namespace-getter created
   rolebinding.rbac.authorization.k8s.io/gke-connect-namespace-getter created
   secret/http-proxy created
   deployment.apps/gke-connect-agent-20210416-01-00 created
   service/gke-connect-monitoring created
   secret/creds-gcp create
   

3. Get the membership resource from the cluster:

   kubectl get memberships membership -o yaml
   

   The output should include the Workoad Identity pool assigned by the fleet, where PROJECT_ID is your project ID:

   workload_identity_pool: PROJECT_ID.svc.id.goog
   

   This means that the cluster registered successfully.

Create certificate authorities to issue certificates

To issue certificates to your Pods, create a CA Service pool and the following certificate authorities (CAs):

• Root CA. This is the root of trust for all issued mesh certificates. You can use an existing root CA if you have one. Create the root CA in the enterprise tier, which is meant for long-lived, low-volume certificate issuance.
• Subordinate CA. This CA issues certificates for workloads. Create the subordinate CA in the region where your cluster is deployed. Create the subordinate CA in the devops tier, which is meant for short-lived, high-volume certificate issuance.

Creating a subordinate CA is optional, but we strongly recommend creating one rather than using your root CA to issue GKE mesh certificates. If you decide to use the root CA to issue mesh certificates, ensure that the default config-based issuance mode remains permitted.

The subordinate CA can be in a different region from your cluster, but we strongly recommend creating it in the same region as your cluster to optimize performance. You can, however, create the root and subordinate CAs in different regions without any impact to performance or availability.

These regions are supported for CA Service:

The list of supported locations can also be checked by running the following command:

gcloud privateca locations list

1. Grant the IAM roles/privateca.caManager to individuals who create a CA pool and a CA. Note that for MEMBER, the correct format is user:userid@example.com. If that person is the current user, you can obtain the current user ID with the shell command $(gcloud auth list --filter=status:ACTIVE --format="value(account)").

   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member=MEMBER \
     --role=roles/privateca.caManager
   

2. Grant the role role/privateca.admin for CA Service to individuals who need to modify IAM policies, where MEMBER is an individual who needs this access, specifically, any individuals who perform the steps that follow that grant the privateca.auditor and privateca.certificateManager roles:

   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member=MEMBER \
     --role=roles/privateca.admin
   

3. Create the root CA Service pool.

   gcloud privateca pools create ROOT_CA_POOL_NAME \
     --location ROOT_CA_POOL_LOCATION \
     --tier enterprise
   

4. Create a root CA.

   gcloud privateca roots create ROOT_CA_NAME --pool ROOT_CA_POOL_NAME \
     --subject "CN=ROOT_CA_NAME, O=ROOT_CA_ORGANIZATION" \
     --key-algorithm="ec-p256-sha256" \
     --max-chain-length=1 \
     --location ROOT_CA_POOL_LOCATION
   

   For this demonstration setup, use the following values for the variables:

   • ROOT_CA_POOL_NAME=td_sec_pool
   • ROOT_CA_NAME=pkcs2-ca
   • ROOT_CA_POOL_LOCATION=us-east1
   • ROOT_CA_ORGANIZATION="TestCorpLLC"

5. Create the subordinate pool and subordinate CA. Ensure that the default config-based issuance mode remains permitted.

   gcloud privateca pools create SUBORDINATE_CA_POOL_NAME \
     --location SUBORDINATE_CA_POOL_LOCATION \
     --tier devops
   

   gcloud privateca subordinates create SUBORDINATE_CA_NAME \
     --pool SUBORDINATE_CA_POOL_NAME \
     --location SUBORDINATE_CA_POOL_LOCATION \
     --issuer-pool ROOT_CA_POOL_NAME \
     --issuer-location ROOT_CA_POOL_LOCATION \
     --subject "CN=SUBORDINATE_CA_NAME, O=SUBORDINATE_CA_ORGANIZATION" \
     --key-algorithm "ec-p256-sha256" \
     --use-preset-profile subordinate_mtls_pathlen_0
   

   For this demonstration setup, use the following values for the variables:

   • SUBORDINATE_CA_POOL_NAME="td-ca-pool"
   • SUBORDINATE_CA_POOL_LOCATION=us-east1
   • SUBORDINATE_CA_NAME="td-ca"
   • SUBORDINATE_CA_ORGANIZATION="TestCorpLLC"
   • ROOT_CA_POOL_NAME=td_sec_pool
   • ROOT_CA_POOL_LOCATION=us-east1

6. Grant the IAM privateca.auditor role for the root CA pool to allow access from the GKE service account:

   gcloud privateca pools add-iam-policy-binding ROOT_CA_POOL_NAME \
    --location ROOT_CA_POOL_LOCATION \
    --role roles/privateca.auditor \
    --member="serviceAccount:service-PROJNUM@container-engine-robot.iam.gserviceaccount.com"
   

7. Grant the IAM privateca.certificateManager role for the subordinate CA pool to allow access from the GKE service account:

   gcloud privateca pools add-iam-policy-binding SUBORDINATE_CA_POOL_NAME \
     --location SUBORDINATE_CA_POOL_LOCATION \
     --role roles/privateca.certificateManager \
     --member="serviceAccount:service-PROJNUM@container-engine-robot.iam.gserviceaccount.com"
   

8. Save the following WorkloadCertificateConfig YAML configuration to tell your cluster how to issue mesh certificates:

   apiVersion: security.cloud.google.com/v1
   kind: WorkloadCertificateConfig
   metadata:
     name: default
   spec:
     # Required. The CA service that issues your certificates.
     certificateAuthorityConfig:
       certificateAuthorityServiceConfig:
         endpointURI: ISSUING_CA_POOL_URI
   
     # Required. The key algorithm to use. Choice of RSA or ECDSA.
     #
     # To maximize compatibility with various TLS stacks, your workloads
     # should use keys of the same family as your root and subordinate CAs.
     #
     # To use RSA, specify configuration such as:
     #   keyAlgorithm:
     #     rsa:
     #       modulusSize: 4096
     #
     # Currently, the only supported ECDSA curves are "P256" and "P384", and the only
     # supported RSA modulus sizes are 2048, 3072 and 4096.
     keyAlgorithm:
       rsa:
         modulusSize: 4096
   
     # Optional. Validity duration of issued certificates, in seconds.
     #
     # Defaults to 86400 (1 day) if not specified.
     validityDurationSeconds: 86400
   
     # Optional. Try to start rotating the certificate once this
     # percentage of validityDurationSeconds is remaining.
     #
     # Defaults to 50 if not specified.
     rotationWindowPercentage: 50
   
   

   Replace the following:

   • The project ID of the project in which your cluster runs:

     PROJECT_ID

   • The fully qualified URI of the CA that issues your mesh certificates (ISSUING_CA_POOL_URI). This can be either your subordinate CA (recommended) or your root CA. The format is:

     //privateca.googleapis.com/projects/PROJECT_ID/locations/SUBORDINATE_CA_POOL_LOCATION/caPools/SUBORDINATE_CA_POOL_NAME

9. Save the following TrustConfig YAML configuration to tell your cluster how to trust the issued certificates:

   apiVersion: security.cloud.google.com/v1
   kind: TrustConfig
   metadata:
     name: default
   spec:
     # You must include a trustStores entry for the trust domain that
     # your cluster is enrolled in.
     trustStores:
     - trustDomain: PROJECT_ID.svc.id.goog
       # Trust identities in this trustDomain if they appear in a certificate
       # that chains up to this root CA.
       trustAnchors:
       - certificateAuthorityServiceURI: ROOT_CA_POOL_URI
   

   Replace the following:

   • The project ID of the project in which your cluster runs:

     PROJECT_ID

   • The fully qualified URI of the root CA pool (ROOT_CA_POOL_URI). The format is:

     //privateca.googleapis.com/projects/PROJECT_ID/locations/ROOT_CA_POOL_LOCATION/caPools/ROOT_CA_POOL_NAME

10. Apply the configurations to your cluster:

    kubectl apply -f WorkloadCertificateConfig.yaml
    kubectl apply -f TrustConfig.yaml
    

Create a proxyless gRPC service with NEGs

For PSM security, you need a proxyless gRPC server capable of using xDS to acquire security configuration from Cloud Service Mesh. This step is similar to Configuring GKE services with NEGs in the PSM load balancing setup guide, but you use the xDS-enabled helloworld server in the xDS example in the grpc-java repository instead of the java-example-hostname image.

You build and run this server in a container built from an openjdk:8-jdk image. You also use the named NEG feature, which lets you specify a name for the NEG. This simplifies later steps because your deployment knows the name of the NEG without having to look it up.

The following is a complete example of the gRPC server Kubernetes spec. Note the following:

• The spec creates a Kubernetes service account example-grpc-server that is used by the gRPC server Pod.
• The spec uses the name field in the cloud.google.com/neg annotation of the service to specify the NEG name example-grpc-server.
• The variable ${PROJNUM} represents the project number of your project.
• The spec uses the initContainers section to run a bootstrap generator to populate the bootstrap file that the proxyless gRPC library needs. This bootstrap file resides at /tmp/grpc-xds/td-grpc-bootstrap.json in the gRPC server container called example-grpc-server.

Add the following annotation to your Pod spec:

 annotations:
   security.cloud.google.com/use-workload-certificates: ""

You can see the correct placement in the full spec that follows.

On creation, each Pod gets a volume at /var/run/secrets/workload-spiffe-credentials. This volume contains the following:

• private_key.pem is an automatically generated private key.
• certificates.pem is a bundle of PEM-formatted certificates that can be presented to another Pod as the client certificate chain, or used as a server certificate chain.
• ca_certificates.pem is a bundle of PEM-formatted certificates to use as trust anchors when validating the client certificate chain presented by another Pod, or the server certificate chain received when connecting to another Pod.

Note that ca_certificates.pem contains certificates for the local trust domain for the workloads, which is the cluster's workload pool.

The leaf certificate in certificates.pem contains the following plain-text SPIFFE identity assertion:

spiffe://WORKLOAD_POOL/ns/NAMESPACE/sa/KUBERNETES_SERVICE_ACCOUNT

In this assertion:

• WORKLOAD_POOL is the name of the cluster workload pool.
• NAMESPACE is the namespace of your Kubernetes service account.
• KUBERNETES_SERVICE_ACCOUNT is the name of your Kubernetes service account.

The following instructions for your language create the spec to use in this example.

1. Apply the spec:

   kubectl apply -f example-grpc-server.yaml
   

2. Grant the required roles to the service account:

   gcloud iam service-accounts add-iam-policy-binding \
     --role roles/iam.workloadIdentityUser \
     --member "serviceAccount:${PROJECT_ID}.svc.id.goog[default/example-grpc-server]" \
     ${PROJNUM}-compute@developer.gserviceaccount.com
   
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
     --member "serviceAccount:${PROJECT_ID}.svc.id.goog[default/example-grpc-server]" \
     --role roles/trafficdirector.client
   

3. Run these commands to verify that the service and Pod are created correctly:

   kubectl get deploy/example-grpc-server
   kubectl get svc/example-grpc-server
   

4. Verify that the NEG name is correct:

   gcloud compute network-endpoint-groups list \
       --filter "name=example-grpc-server" --format "value(name)"
   

   The command should return the NEG name example-grpc-server.

Configure Cloud Service Mesh with Google Cloud load balancing components

The steps in this section are similar to those in Configuring Cloud Service Mesh with load balancing components, but there are some changes, as described in the follow sections.

Create the health check, firewall rule, and backend service

When the gRPC server is configured to use mTLS, gRPC health checks don't work because the health checking client cannot present a valid client certificate to the servers. You can address this in one of two ways.

In the first approach, you have the server create an additional serving port that is designated as the health checking port. This is attached to a special health check service, as plain text or TLS to that port.

The xDS helloworld example server uses PORT_NUMBER + 1 as the plain text health checking port. The example uses 50052 as the health checking port because 50051 is the gRPC application server port.

In the second approach, you configure health checking to check only TCP connectivity to the application serving port. This checks only connectivity, and it also generates unnecessary traffic to the server when there are unsuccessful TLS handshakes. For this reason, we recommend that you use the first approach.

1. Create the health check. Note that health checking does not start until you create and start the server.

   • If you are creating a designated serving port for health checking, which is the approach we recommend, use this command:

     gcloud compute health-checks create grpc grpc-gke-helloworld-hc \
      --enable-logging --port 50052
     

   • If you are creating a TCP health check, which we don't recommend, use this command:

     gcloud compute health-checks create tcp grpc-gke-helloworld-hc \
     --use-serving-port
     

2. Create the firewall. Ensure that the value of --target-tags matches the value you provided for --tags in the section Create or update a GKE cluster.

   gcloud compute firewall-rules create grpc-gke-allow-health-checks \
     --network default --action allow --direction INGRESS \
     --source-ranges 35.191.0.0/16,130.211.0.0/22 \
     --target-tags allow-health-checks \
     --rules tcp:50051-50052
   

3. Create the backend service:

   gcloud compute backend-services create grpc-gke-helloworld-service \
      --global \
      --load-balancing-scheme=INTERNAL_SELF_MANAGED \
      --protocol=GRPC \
      --health-checks grpc-gke-helloworld-hc
   

4. Attach the NEG to the backend service:

   gcloud compute backend-services add-backend grpc-gke-helloworld-service \
      --global \
      --network-endpoint-group example-grpc-server \
      --network-endpoint-group-zone ${ZONE} \
      --balancing-mode RATE \
      --max-rate-per-endpoint 5
   

Create the Mesh and GRPCRoute resources

This is similar to how you set up the Mesh and GRPCRoute resources in Set up proxyless gRPC services.

1. Create the Mesh specification and save it in a file called mesh.yaml.

   name: grpc-mesh
   

2. Import the Mesh resource from the specification.

   gcloud network-services meshes import grpc-mesh \
     --source=mesh.yaml \
     --location=global
   

3. Create the GRPCRoute specification and save it in a file called grpc_route.yaml.

   name: helloworld-grpc-route
   hostnames:
   - helloworld-gke:8000
   meshes:
   - projects/PROJECT_NUMBER/locations/global/meshes/grpc-mesh
   rules:
   - action:
       destinations:
       - serviceName: projects/PROJECT_NUMBER/locations/global/backendServices/grpc-gke-helloworld-service
   

4. Import the GRPCRoute resource from the grpc_route.yaml specification.

   gcloud network-services grpc-routes import helloworld-grpc-route \
     --source=grpc_route.yaml \
     --location=global
   

Configure Cloud Service Mesh with proxyless gRPC Security

This example demonstrates how to configure mTLS on the client and server sides.

Format for policy references

Note the following required format for referring to server TLS and client TLS policies:

projects/PROJECT_ID/locations/global/[serverTlsPolicies|clientTlsPolicies]/[server-tls-policy|client-mtls-policy]

For example:

projects/PROJECT_ID/locations/global/serverTlsPolicies/server-tls-policy

projects/PROJECT_ID/locations/global/clientTlsPolicies/client-mtls-policy

Configure mTLS on the server side

First, you create a server TLS policy. The policy asks the gRPC server side to use the certificateProvicerInstance plugin config identified by the name google_cloud_private_spiffe for the identity certificate, which is part of the serverCertificate. The mtlsPolicy section indicates mTLS security and uses the same google_cloud_private_spiffe as the plugin config for clientValidationCa, which is the root (validation) certificate specification.

Next, you create an endpoint policy. This specifies that a backend, for example a gRPC server, using port 50051 with any or no metadata labels, receives the attached server TLS policy named server-mtls-policy. You specify metadata labels using MATCH_ALL or a supported value. The supported metadata labels can be found in field endpointMatcher.metadataLabelMatcher.metadataLabelMatchCriteria in NetworkServicesEndpointPolicy Document. You create the endpoint policy with a temporary file ep-mtls-psms.yaml that contains the values for the endpoint policy resource using the policy that you already defined.

1. Create a temporary file server-mtls-policy.yaml in the current directory with the values of the server TLS policy resource:

   name: "projects/PROJECT_ID/locations/global/serverTlsPolicies/server-mtls-policy"
   serverCertificate:
     certificateProviderInstance:
       pluginInstance: google_cloud_private_spiffe
   mtlsPolicy:
     clientValidationCa:
     - certificateProviderInstance:
         pluginInstance: google_cloud_private_spiffe
   

2. Create a server TLS policy resource called server-mtls-policy by importing the temporary file server-mtls-policy.yaml:

   gcloud network-security server-tls-policies import server-mtls-policy \
     --source=server-mtls-policy.yaml --location=global
   

3. Create the endpoint policy by creating the temporary file ep-mtls-psms.yaml:

   name: "ep-mtls-psms"
   type: "GRPC_SERVER"
   serverTlsPolicy: "projects/PROJECT_ID/locations/global/serverTlsPolicies/server-mtls-policy"
   trafficPortSelector:
     ports:
     - "50051"
   endpointMatcher:
     metadataLabelMatcher:
       metadataLabelMatchCriteria: "MATCH_ALL"
       metadataLabels:
       - labelName: app
         labelValue: helloworld
   

4. Create the endpoint policy resource by importing the file ep-mtls-psms.yaml:

   gcloud beta network-services endpoint-policies import ep-mtls-psms \
     --source=ep-mtls-psms.yaml --location=global
   

Configure mTLS on the client side

The client-side security policy is attached to the backend service. When a client accesses a backend (the gRPC server) through the backend service, the attached client-side security policy is sent to the client.

1. Create the client TLS policy resource contents in a temporary file called client-mtls-policy.yaml in the current directory:

   name: "client-mtls-policy"
   clientCertificate:
     certificateProviderInstance:
       pluginInstance: google_cloud_private_spiffe
   serverValidationCa:
   - certificateProviderInstance:
       pluginInstance: google_cloud_private_spiffe
   

2. Create the client TLS policy resource called client-mtls-policy by importing the temporary file client-mtls-policy.yaml:

   gcloud network-security client-tls-policies import client-mtls-policy \
     --source=client-mtls-policy.yaml --location=global
   

3. Create a snippet in a temporary file to reference this policy and add details for subjectAltNames in the SecuritySettings message as in the following example. Replace ${PROJECT_ID} with your project ID value, which is the value of the ${PROJECT_ID} environment variable described previously. Note that example-grpc-server in subjectAltNames is the Kubernetes service account name that is used for the gRPC server Pod in the deployment spec.

   if [ -z "$PROJECT_ID" ] ; then echo Please make sure PROJECT_ID is set. ; fi
   cat << EOF > client-security-settings.yaml
   securitySettings:
     clientTlsPolicy: projects/${PROJECT_ID}/locations/global/clientTlsPolicies/client-mtls-policy
     subjectAltNames:
       - "spiffe://${PROJECT_ID}.svc.id.goog/ns/default/sa/example-grpc-server"
   EOF
   

4. Add the securitySettings message to the backend service you already created. These steps export the current backend service contents, add the client securitySetting message and re-importing the new content to update the backend service.

   gcloud compute backend-services export grpc-gke-helloworld-service --global \
     --destination=/tmp/grpc-gke-helloworld-service.yaml
   
   cat /tmp/grpc-gke-helloworld-service.yaml client-security-settings.yaml \
     >/tmp/grpc-gke-helloworld-service1.yaml
   
   gcloud compute backend-services import grpc-gke-helloworld-service --global \
     --source=/tmp/grpc-gke-helloworld-service1.yaml -q
   

Verify the configuration

Cloud Service Mesh configuration is now complete, including server- and client-side security. Next, you prepare and run the server and client workloads. This completes the example.

Create a proxyless gRPC client

This step is similar to the previous step Creating a proxyless gRPC service. You use the xDS-enabled helloworld client from the xDS example directory in the grpc-java repository. You build and run the client in a container built from an openjdk:8-jdk image. The gRPC client Kubernetes spec does the following.

• It creates a Kubernetes service account example-grpc-client that is used by the gRPC client Pod.
• ${PROJNUM} represents the project number of your project and needs to be replaced with the actual number.

Add the following annotation to your Pod spec:

  annotations:
    security.cloud.google.com/use-workload-certificates: ""

On creation, each Pod gets a volume at /var/run/secrets/workload-spiffe-credentials. This volume contains the following:

• private_key.pem is an automatically generated private key.
• certificates.pem is a bundle of PEM-formatted certificates that can be presented to another Pod as the client certificate chain, or used as a server certificate chain.
• ca_certificates.pem is a bundle of PEM-formatted certificates to use as trust anchors when validating the client certificate chain presented by another Pod, or the server certificate chain received when connecting to another Pod.

Note that ca_certificates.pem contains the root certificates for the local trust domain for the workloads, which is the cluster's workload pool.

The leaf certificate in certificates.pem contains the following plain-text SPIFFE identity assertion:

spiffe://WORKLOAD_POOL/ns/NAMESPACE/sa/KUBERNETES_SERVICE_ACCOUNT

In this assertion:

• WORKLOAD_POOL is the name of the cluster workload pool.
• NAMESPACE is the name of your Kubernetes service account.
• KUBERNETES_SERVICE_ACCOUNT is the namespace of your Kubernetes service account.

The following instructions for your language create the spec to use in this example.

Complete the process as follows.

1. Apply the spec:

   kubectl apply -f example-grpc-client.yaml
   

2. Grant the required roles to the service account:

   gcloud iam service-accounts add-iam-policy-binding \
     --role roles/iam.workloadIdentityUser \
     --member "serviceAccount:${PROJECT_ID}.svc.id.goog[default/example-grpc-client]" \
     ${PROJNUM}-compute@developer.gserviceaccount.com
   
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
     --member "serviceAccount:${PROJECT_ID}.svc.id.goog[default/example-grpc-client]" \
     --role roles/trafficdirector.client
   

3. Verify that the client Pod is running:

   kubectl get pods
   

   The command returns text similar to the following:

   NAMESPACE   NAME                                    READY   STATUS    RESTARTS   AGE
   default     example-grpc-client-7c969bb997-9fzjv    1/1     Running   0          104s
   [..skip..]
   

Run the server

Build and run the xDS-enabled helloworld server in the server Pod you created earlier.

The health checking process takes from 3 to 5 minutes to show that your service is healthy after the server starts.

Run the client and verify the configuration

Build and run the xDS-enabled helloworld client in the client Pod you created earlier.

Configure service-level access with an authorization policy

gRFC A41 support is required for authorization policy support. You can find the required language versions on github

Use these instructions to configure service-level access with authorization policies. Before you create authorization policies, read the caution in Restrict access using authorization.

To make it easier to verify the configuration, create an additional hostname that the client can use to refer to the helloworld-gke service.

1. Update the GRPCRoute specification previously stored in grpc_route.yaml

   name: helloworld-grpc-route
   hostnames:
   - helloworld-gke:8000
   - helloworld-gke-noaccess:8000
   meshes:
   - projects/PROJECT_NUMBER/locations/global/meshes/grpc-mesh
   rules:
   - action:
       destinations:
       - serviceName: projects/PROJECT_NUMBER/locations/global/backendServices/grpc-gke-helloworld-service
   

2. Import the GRPCRoute resource again from the grpc_route.yaml specification.

   gcloud network-services grpc-routes import helloworld-grpc-route \
     --source=grpc_route.yaml \
     --location=global
   

The following instructions create an authorization policy that allows requests that are sent by the example-grpc-client account in which the hostname is helloworld-gke:8000 and the port is 50051.

Validate the authorization policy

Use these instructions to confirm that the authorization policy is working correctly.

Use TLS instead of mTLS

Using TLS in this example requires only a small change.

1. In the ServerTlsPolicy, drop the mtlsPolicy:

   cat << EOF > server-tls-policy.yaml
   name: "server-tls-policy"
   serverCertificate:
     certificateProviderInstance:
       pluginInstance: google_cloud_private_spiffe
   EOF
   

2. Use this policy in the EndpointPolicy instead:

   cat << EOF > ep-tls-psms.yaml
   name: "ep-mtls-psms"
   type: "GRPC_SERVER"
   serverTlsPolicy: "projects/${PROJECT_ID}/locations/global/serverTlsPolicies/server-tls-policy"
   trafficPortSelector:
     ports:
     - "50051"
   endpointMatcher:
     metadataLabelMatcher:
       metadataLabelMatchCriteria: "MATCH_ALL"
       metadataLabels: []
   EOF
   

3. The ClientTlsPolicy for mTLS also works in the TLS case but the clientCertificate section of the policy can be dropped since it is not required for TLS:

   cat << EOF > client-tls-policy.yaml
   name: "client-tls-policy"
   serverValidationCa:
   - certificateProviderInstance:
       pluginInstance: google_cloud_private_spiffe
   EOF
   

Use service security with the Wallet example

This section provides a high-level overview of how to enable the Wallet example with service security, for Java, C++, and Go.

Bootstrap file

The setup process in this guide uses a bootstrap generator to create the required bootstrap file. This section provides reference information about the bootstrap file itself.

The bootstrap file contains configuration information required by proxyless gRPC code, including connection information for the xDS server. The bootstrap file contains security configuration that is required by the proxyless gRPC security feature. The gRPC server requires one additional field. A sample bootstrap file looks like this:

{
  "xds_servers": [
    {
      "server_uri": "trafficdirector.googleapis.com:443",
      "channel_creds": [
        {
          "type": "google_default"
        }
      ],
      "server_features": [
        "xds_v3"
      ]
    }
  ],
  "authorities": {
    "traffic-director-c2p.xds.googleapis.com": {
      "xds_servers": [
        {
          "server_uri": "dns:///directpath-pa.googleapis.com",
          "channel_creds": [
            {
              "type": "google_default"
            }
          ],
          "server_features": [
            "xds_v3",
            "ignore_resource_deletion"
          ]
        }
      ],
      "client_listener_resource_name_template": "xdstp://traffic-director-c2p.xds.googleapis.com/envoy.config.listener.v3.Listener/%s"
    }
  },
  "node": {
    "id": "projects/9876012345/networks/mesh:grpc-mesh/nodes/b59f49cc-d95a-4462-9126-112f794d5dd3",
    "cluster": "cluster",
    "metadata": {
      "INSTANCE_IP": "10.28.2.8",
      "TRAFFICDIRECTOR_DIRECTPATH_C2P_IPV6_CAPABLE": true,
      "TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "223606568246",
      "TRAFFICDIRECTOR_NETWORK_NAME": "default",
      "app": "helloworld"
    },
    "locality": {
      "zone": "us-central1-c"
    }
  },
  "certificate_providers": {
    "google_cloud_private_spiffe": {
      "plugin_name": "file_watcher",
      "config": {
        "certificate_file": "/var/run/secrets/workload-spiffe-credentials/certificates.pem",
        "private_key_file": "/var/run/secrets/workload-spiffe-credentials/private_key.pem",
        "ca_certificate_file": "/var/run/secrets/workload-spiffe-credentials/ca_certificates.pem",
        "refresh_interval": "600s"
      }
    }
  },
  "server_listener_resource_name_template": "grpc/server?xds.resource.listening_address=%s"
}

Updates to the bootstrap file for security service

The following fields reflect modifications related to security and xDS v3 usage:

The id field inside node provides a unique identity for the gRPC client to Cloud Service Mesh. You must provide the Google Cloud project number and network name using the node ID in the this format:

projects/{project number}/networks/{network name}/nodes/[UNIQUE_ID]

An example for project number 1234 and the default network is:

projects/1234/networks/default/nodes/client1

The INSTANCE_IP field is the IP address of the Pod, or 0.0.0.0 to indicate INADDR_ANY. This field is used by the gRPC server for fetching the Listener resource from Cloud Service Mesh for server side security.

Security config fields in the bootstrap file

The contents of the config JSON structure for the file_watcher plugin are:

• certificate_file: Required string. This value is the location of the identity certificate.
• private_key_file: Required string. The value is the location of the private key file, which should match the identity certificate.
• ca_certificate_file: Required string. The value is the location of the root certificate, which is also know as the trust bundle.
• refresh_interval: Optional string. The value indicates the refresh interval, specified using the string representation of a Duration's JSON mapping. The default value is "600s", a duration of 10 minutes.

Bootstrap generator

The bootstrap generator container image is available at gcr.io/trafficdirector-prod/td-grpc-bootstrap:0.16.0. Its source code is available at at https://github.com/GoogleCloudPlatform/traffic-director-grpc-bootstrap. The most commonly used command line options are these:

• --output: Use this option to specify where the output bootstrap file is written to, for example, the command --output /tmp/bootstrap/td-grpc-bootstrap.json generates the bootstrap file to /tmp/bootstrap/td-grpc-bootstrap.json in the Pod's file system.
• --config-mesh-experimental: Use this option to specify the mesh name, matching the Mesh resource.
• --node-metadata: Use this flag to populate the node metadata in the bootstrap file. This is required when you use metadata label matchers in the EndpointPolicy where Cloud Service Mesh uses the label data provided in the node metadata section of the bootstrap file. The argument is supplied in the form key=value, for example: --node-metadata version=prod --node-metadata type=grpc

The previous information adds the following in the node metadata section of the bootstrap file:

{
  "node": {
...
    "metadata": {
      "version": "prod",
      "type": "grpc",
...
    },
...
  },
...
}

Delete the deployment

You can optionally run these commands to delete the deployment you created using this guide.

To delete the cluster, run this command:

gcloud container clusters delete CLUSTER_NAME --zone ZONE --quiet

To delete the resources you created, run these commands:

gcloud compute backend-services delete grpc-gke-helloworld-service --global --quiet
gcloud compute network-endpoint-groups delete example-grpc-server --zone ZONE --quiet
gcloud compute firewall-rules delete grpc-gke-allow-health-checks --quiet
gcloud compute health-checks delete grpc-gke-helloworld-hc --quiet
gcloud network-services endpoint-policies delete ep-mtls-psms \
    --location=global --quiet
gcloud network-security authorization-policies delete helloworld-gke-authz-policy \
   --location=global --quiet
gcloud network-security client-tls-policies delete client-mtls-policy \
    --location=global --quiet
gcloud network-security server-tls-policies delete server-tls-policy \
    --location=global --quiet
gcloud network-security server-tls-policies delete server-mtls-policy \
    --location=global --quiet

Troubleshooting

Use these instructions to help you resolve problems with your security deployment.

Workloads are unable to get config from Cloud Service Mesh

If you see an error similar to this:

PERMISSION_DENIED: Request had insufficient authentication scopes.

Make sure of the following:

• You created your GKE cluster with the argument --scopes=cloud-platform argument.
• You assigned the roles/trafficdirector.client to your Kuberneters service accounts.
• You assigned the roles/trafficdirector.client to your default Google Cloud service account (${GSA_EMAIL} above).
• You enabled the trafficdirector.googleapis.com service (API).

Your gRPC server does not use TLS/mTLS even with correct Cloud Service Mesh configuration

Make sure you specify GRPC_SERVER in your endpoint policies configuration. If you specified SIDECAR_PROXY gRPC ignores the configuration.

You are unable to create the GKE cluster with the requested cluster version

The GKE cluster creation command might fail with an error something like this:

Node version "1.20.5-gke.2000" is unsupported.

Make sure that you are using the argument --release-channel rapid in your cluster creation command. You need to use the rapid release channel to get the correct version for this release.

You see a No usable endpoint error

If a client can't communicate with the server because of a No usable endpoint error, the health checker might have marked the server backends as unhealthy. To check the health of the backends, run this gcloud command:

gcloud compute backend-services get-health grpc-gke-helloworld-service --global

If the command returns the backend status unhealthy, it might be for one of these reasons:

• The firewall was not created or does not contain the correct source IP range.
• The target tags on your firewall don't match the tags on the cluster you created.

Workloads are unable to communicate in the security setup

If your workloads are not able to communicate after you set up security for your proxyless service mesh, follow these instructions to determine the cause.

1. Disable proxyless security and eliminate issues in the proxyless service mesh load balancing use-cases. To disable security in the mesh do one of the following:
   1. Use plain text credentials on the client and server side OR
   2. don't configure security for the backend service and endpoint policy in the Cloud Service Mesh configuration.

Follow the steps in Troubleshooting proxyless Cloud Service Mesh deployments, because there is no security setup in your deployment.

1. Modify your workloads to use xDS credentials with plain text or insecure credentials as the fallback credentials. Keep the Cloud Service Mesh configuration with security disabled as discussed previously. In this case, although gRPC is allowing Cloud Service Mesh to configure security, Cloud Service Mesh does not send security information in which case gRPC should fall back to plain text (or insecure) credentials which should work similar to the previous first case. If this case does not work, do the following:

   1. Increase the logging level on both the client and server side so that you can see the xDS messages exchanged between gRPC and Cloud Service Mesh.
   2. Ensure that Cloud Service Mesh does not have security enabled in the CDS and LDS responses that are sent to the workloads.
   3. Ensure that the workloads are not using TLS or mTLS modes in their channels. If you see any log messages related to TLS handshakes, check your application source code and make sure that you are using insecure or plain text as your fallback credentials. If the application source code is correct, this might be a bug in the gRPC library

2. Verify that the CA Service integration with GKE is working correctly for your GKE cluster by following the troubleshooting steps in that User Guide. Make sure that the certificates and keys provided by that feature are made available in the specified directory, /var/run/secrets/workload-spiffe-credentials/.

3. Enable TLS (instead of mTLS) in your mesh, as described previously, and restart your client and server workloads.

   1. Increase the logging level on both the client and server side to be able to see the xDS messages exchanged between gRPC and Cloud Service Mesh.
   2. Ensure that Cloud Service Mesh has enabled security in the CDS and LDS responses that are sent to the workloads.

Client fails with a CertificateException and a message Peer certificate SAN check failed

This indicates a problem with the subjectAltNames values in the SecuritySettings message. Note that these values are based on the Kubernetes services you created for your backend service. For every such Kubernetes service you created, there is an associated SPIFFE ID, in this format:

spiffe://${WORKLOAD_POOL}/ns/${K8S_NAMESPACE}/sa/${SERVICE_ACCOUNT}

These values are:

• WORKLOAD_POOL: The workload pool for the cluster, which is ${PROJECT_ID}.svc.id.goog
• K8S_NAMESPACE: The Kubernetes namespace you used in the deployment of the service
• SERVICE_ACCOUNT: The Kubernetes service account you used in the deployment of the service

For every Kubernetes service you attached to your backend service as a network endpoint group, make sure that you correctly computed the SPIFFE ID and added that SPIFFE ID to the subjectAltNames field in the SecuritySettings message.

Applications cannot use the mTLS certificates with your gRPC library

If your applications are unable to use the mTLS certificates with your gRPC library, do the following:

1. Verify that the Pod spec contains the security.cloud.google.com/use-workload-certificates annotation that is described in Creating a proxyless gRPC service with NEGs.

2. Verify that the files containing the certificate chain along with the leaf certificate, private key, and the trusted CA certificates are accessible at the following paths from within the Pod:

   1. Certificate chain along with leaf cert: "/var/run/secrets/workload-spiffe-credentials/certificates.pem"
   2. Private key: "/var/run/secrets/workload-spiffe-credentials/private_key.pem"
   3. CA Bundle: "/var/run/secrets/workload-spiffe-credentials/ca_certificates.pem"

3. If the certificates in the previous step are not available, do the following:

     gcloud privateca subordinates describe SUBORDINATE_CA_POOL_NAME 
   
       --location=LOCATION
     

   1. Verify that GKE's control plane has the correct IAM role binding, granting it access to CA Service:

      # Get the IAM policy for the CA
      gcloud privateca roots get-iam-policy ROOT_CA_POOL_NAME
      
      # Verify that there is an IAM binding granting access in the following format
      - members:
      - serviceAccount:service-projnumber@container-engine-robot.iam.gserviceaccount.com
      role: roles/privateca.certificateManager
      
      # Where projnumber is the project number (e.g. 2915810291) for the GKE cluster.
      

   2. Verify that the certificate has not expired. This is the certificate chain and leaf certificate at /var/run/secrets/workload-spiffe-credentials/certificates.pem. To check, run this command:

      cat /var/run/secrets/workload-spiffe-credentials/certificates.pem | openssl x509 -text -noout | grep "Not After"
      

   3. Verify that the key type is supported by your application by running this command:

      cat /var/run/secrets/workload-spiffe-credentials/certificates.pem | openssl x509 -text -noout | grep "Public Key Algorithm" -A 3
      

   4. Verify that your gRPC Java application has the following keyAlgorithm in the WorkloadCertificateConfig YAML file:

     keyAlgorithm:
       rsa:
         modulusSize: 4096
   

4. Verify that the CA uses the same key family as the certificate key.

An application's certificate is rejected by the client, server, or peer

1. Verify that the peer application uses the same trust bundle to verify the certificate.
2. Verify that the certificate in use is not expired (certificate chain along with leaf cert: "/var/run/secrets/workload-spiffe-credentials/certificates.pem").

Pods remain in a pending state

If the Pods stay in a pending state during the setup process, increase the CPU and memory resources for the Pods in your deployment spec.

Unable to create cluster with the --enable-mesh-certificates flag

Ensure that you are running the latest version of the gcloud CLI:

gcloud components update

Note that the --enable-mesh-certificates flag works only with gcloud beta.

Pods don't start

Pods that use GKE mesh certificates might fail to start if certificate provisioning is failing. This can happen in situations like the following:

• The WorkloadCertificateConfig or the TrustConfig is misconfigured or missing.
• CSRs aren't being approved.

You can check whether certificate provisioning is failing by checking the Pod events.

1. Check the status of your Pod:

   kubectl get pod -n POD_NAMESPACE POD_NAME
   

   Replace the following:

   • POD_NAMESPACE: the namespace of your Pod.
   • POD_NAME: the name of your Pod.

2. Check recent events for your Pod:

   kubectl describe pod -n POD_NAMESPACE POD_NAME
   

3. If certificate provisioning is failing, you will see an event with Type=Warning, Reason=FailedMount, From=kubelet, and a Message field that begins with MountVolume.SetUp failed for volume "gke-workload-certificates". The Message field contains troubleshooting information.

   Events:
     Type     Reason       Age                From       Message
     ----     ------       ----               ----       -------
     Warning  FailedMount  13s (x7 over 46s)  kubelet    MountVolume.SetUp failed for volume "gke-workload-certificates" : rpc error: code = Internal desc = unable to mount volume: store.CreateVolume, err: unable to create volume "csi-4d540ed59ef937fbb41a9bf5380a5a534edb3eedf037fe64be36bab0abf45c9c": caPEM is nil (check active WorkloadCertificateConfig)
   

4. See the following troubleshooting steps if the reason your Pods don't start is because of misconfigured objects, or because of rejected CSRs.

WorkloadCertificateConfig or TrustConfig is misconfigured

Ensure that you created the WorkloadCertificateConfig and TrustConfig objects correctly. You can diagnose misconfigurations on either of these objects using kubectl.

1. Retrieve the current status.

   For WorkloadCertificateConfig:

   kubectl get WorkloadCertificateConfig default -o yaml
   

   For TrustConfig:

   kubectl get TrustConfig default -o yaml
   

2. Inspect the status output. A valid object will have a condition with type: Ready and status: "True".

   status:
     conditions:
     - lastTransitionTime: "2021-03-04T22:24:11Z"
       message: WorkloadCertificateConfig is ready
       observedGeneration: 1
       reason: ConfigReady
       status: "True"
       type: Ready
   

   For invalid objects, status: "False" appears instead. Thereasonand message field contain additional troubleshooting details.

CSRs are not approved

If something goes wrong during the CSR approval process, you can check the error details in the type: Approved and type: Issued conditions of the CSR.

1. List relevant CSRs using kubectl:

   kubectl get csr \
     --field-selector='spec.signerName=spiffe.gke.io/spiffe-leaf-signer'
   

2. Choose a CSR that is either Approved and not Issued, or is not Approved.

3. Get details for the selected CSR using kubectl:

   kubectl get csr CSR_NAME -o yaml
   

   Replace CSR_NAME with the name of the CSR you chose.

A valid CSR has a condition with type: Approved and status: "True", and a valid certificate in the status.certificate field:

status:
  certificate: <base64-encoded data>
  conditions:
  - lastTransitionTime: "2021-03-04T21:58:46Z"
    lastUpdateTime: "2021-03-04T21:58:46Z"
    message: Approved CSR because it is a valid SPIFFE SVID for the correct identity.
    reason: AutoApproved
    status: "True"
    type: Approved

Troubleshooting information for invalid CSRs appears in the message and reason fields.

Pods are missing certificates

1. Get the Pod spec for your Pod:

   kubectl get pod -n POD_NAMESPACE POD_NAME -o yaml
   

   Replace the following:

   • POD_NAMESPACE: the namespace of your Pod.
   • POD_NAME: the name of your Pod.

2. Verify that the Pod spec contains the security.cloud.google.com/use-workload-certificates annotation described in Configure Pods to receive mTLS credentials.

3. Verify that the GKE mesh certificates admission controller successfully injected a CSI driver volume of type workloadcertificates.security.cloud.google.com into your Pod spec:

   volumes:
   ...
   -csi:
     driver: workloadcertificates.security.cloud.google.com
     name: gke-workload-certificates
   ...
   

4. Check for the presence of a volume mount in each of the containers:

   containers:
   - name: ...
     ...
     volumeMounts:
     - mountPath: /var/run/secrets/workload-spiffe-credentials
       name: gke-workload-certificates
       readOnly: true
     ...
   

5. Verify that the following certificate bundles and the private key are available at the following locations in the Pod:

   • Certificate chain bundle: /var/run/secrets/workload-spiffe-credentials/certificates.pem
   • Private key: /var/run/secrets/workload-spiffe-credentials/private_key.pem
   • CA trust anchor bundle: /var/run/secrets/workload-spiffe-credentials/ca_certificates.pem

6. If the files are not available, perform the following steps:

   1. Retrieve the CA Service (Preview) instance for the cluster:

      kubectl get workloadcertificateconfigs default -o jsonpath '{.spec.certificateAuthorityConfig.certificateAuthorityServiceConfig.endpointURI}'
      

   2. Retrieve the status of the CA Service (Preview) instance:

      gcloud privateca ISSUING_CA_TYPE describe ISSUING_CA_NAME \
        --location ISSUING_CA_LOCATION
      

      Replace the following:

      • ISSUING_CA_TYPE: the issuing CA type, which must be either subordinates or roots.
      • ISSUING_CA_NAME: the name of the issuing CA.
      • ISSUING_CA_LOCATION: the region of the issuing CA.

   3. Get the IAM policy for the root CA:

      gcloud privateca roots get-iam-policy ROOT_CA_NAME
      

      Replace ROOT_CA_NAME with the name of your root CA.

   4. In the IAM policy, verify that the privateca.auditor policy binding exists:

      ...
      - members:
        - serviceAccount:service-PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com
        role: roles/privateca.auditor
      ...
      

      In this example, PROJECT_NUMBER is your cluster's project number.

   5. Get the IAM policy for the subordinate CA:

      gcloud privateca subordinates get-iam-policy SUBORDINATE_CA_NAME
      

      Replace SUBORDINATE_CA_NAME with the subordinate CA name.

   6. In the IAM policy, verify that the privateca.certificateManager policy binding exists:

      ...
      - members:
        - serviceAccount: service-PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com
        role: roles/privateca.certificateManager
      ...
      

      In this example, PROJECT_NUMBER is your cluster's project number.

Applications cannot use issued mTLS credentials

1. Verify that the certificate has not expired:

   cat /var/run/secrets/workload-spiffe-credentials/certificates.pem | openssl x509 -text -noout | grep "Not After"
   

2. Check that the key type you used is supported by your application.

   cat /var/run/secrets/workload-spiffe-credentials/certificates.pem | openssl x509 -text -noout | grep "Public Key Algorithm" -A 3
   

3. Check that the issuing CA uses the same key family as the certificate key.

   1. Get the status of the CA Service (Preview) instance:

      gcloud privateca ISSUING_CA_TYPE describe ISSUING_CA_NAME \
        --location ISSUING_CA_LOCATION
      

      Replace the following:

      • ISSUING_CA_TYPE: the issuing CA type, which must be either subordinates or roots.
      • ISSUING_CA_NAME: the name of the issuing CA.
      • ISSUING_CA_LOCATION: the region of the issuing CA.

   2. Check that the keySpec.algorithm in the output is the same key algorithm you defined in the WorkloadCertificateConfig YAML manifest. The output looks like this:

      config:
        ...
        subjectConfig:
          commonName: td-sub-ca
          subject:
            organization: TestOrgLLC
          subjectAltName: {}
      createTime: '2021-05-04T05:37:58.329293525Z'
      issuingOptions:
        includeCaCertUrl: true
      keySpec:
        algorithm: RSA_PKCS1_2048_SHA256
       ...
      

Certificates get rejected

1. Verify that the peer application uses the same trust bundle to verify the certificate.

2. Verify that the certificate has not expired:

   cat /var/run/secrets/workload-spiffe-credentials/certificates.pem | openssl x509 -text -noout | grep "Not After"
   

3. Verify that the client code, if not using the gRPC Go Credentials Reloading API, periodically refreshes the credentials from the file system.

4. Verify that your workloads are in the same trust domain as your CA. GKE mesh certificates supports communication between workloads in a single trust domain.

Limitations

Cloud Service Mesh service security is supported only with GKE. You cannot deploy service security with Compute Engine.

Cloud Service Mesh does not support scenarios where there are two or more endpoint policy resources that match equally to an endpoint, for example, two policies with the same labels and ports, or two or more policies with different labels that match equally with an endpoint's labels. For more information on how endpoint policys are matched to an endpoint's labels, see the APIs for EndpointPolicy.EndpointMatcher.MetadataLabelMatcher. In such situations, Cloud Service Mesh does not generate security configuration from any of the conflicting policies.