Rotating service account keys

Stay organized with collections Save and categorize content based on your preferences.

This page describes how to rotate keys for the following service accounts:

• Component access
• Connect-register
• Logging-monitoring
• Audit logging
• Usage metering

To rotate your service account keys:

1. Create a directory to store a backup of your current secrets:

   mkdir backup

2. Note the following information for the relevant service account:

   Component access

   Cluster	Secret	Namespace
   Admin	admin-cluster-creds	kube-system
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt
   Admin	private-registry-creds	kube-system
   User	private-registry-creds	kube-system

   • If you aren't using a private registry, the private-registry-creds Secret holds the key for your component access service account.
   • If you are using a private registry, the private-registry-creds Secret holds the credentials for your private registry, not the component access service account key.

   Connect-register

   Cluster	Secret	Namespace
   Admin	admin-cluster-creds	kube-system
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt

   Logging-monitoring

   Cluster	Secret	Namespace
   Admin	admin-cluster-creds	kube-system
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt
   User	google-cloud-credentials	kube-system
   User	stackdriver-service-account-key	knative-serving

   Audit logging

   Cluster	Secret	Namespace
   Admin	admin-cluster-creds	kube-system
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt
   Admin	kube-apiserver	CLUSTER_NAME

   Usage Metering

   Cluster	Secret	Namespace
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt
   User	usage-metering-bigquery-service-account-key	kube-system

   Stackdriver

   Cluster	Secret	Namespace
   Admin	admin-cluster-creds	kube-system
   Admin	user-cluster-creds	CLUSTER_NAME-gke-onprem-mgmt
   User	google-cloud-credentials	kube-system
   User	stackdriver-service-account-key	knative-serving

3. Create a backup of each secret using the following command:

   kubectl get secret SECRET --namespace NAMESPACE \
       --kubeconfig KUBECONFIG -o json > backup/SECRET-NAMESPACE.json

   Replace the following:

   • NAMESPACE: the namespace where the secret is located. For example, kube-system.
   • KUBECONFIG: the path to the kubeconfig file for the admin or user cluster.
   • SECRET: the name of the secret. For example, admin-cluster-creds.

   For example, run the following commands for the audit logging service account:

   kubectl get secret admin-cluster-creds --namespace kube-system \
           --kubeconfig KUBECONFIG -o json > backup/admin-cluster-creds-kube-system.json
   
   kubectl get secret user-cluster-creds --namespace NAMESPACE \
           --kubeconfig KUBECONFIG -o json > backup/user-cluster-creds-NAMESPACE.json
   
   kubectl get secret kube-apiserver --namespace NAMESPACE \
           --kubeconfig KUBECONFIG -o json > backup/kube-apiserver-NAMESPACE.json

4. To create a new service account key file, run the following command:

   gcloud iam service-accounts keys create NEW_KEY_FILE --iam-account IAM_ACCOUNT

   Replace the following:

   • NEW_KEY_FILE: the name for your new service account key file
   • IAM_ACCOUNT: the email address of the service account

5. In the admin cluster configuration file, find the componentAccessServiceAccountKeyPath field, the gkeConnect section, the stackdriver section, and the cloudAuditLogging section. In those places, replace the paths to the service account key files.

6. In the user cluster configuration file, find the componentAccessServiceAccountKeyPath field, the gkeConnect section, the stackdriver section, the cloudAudigLogging section, and the usageMetering section. In those places, replace the paths to the service account key files.

7. Save the changes you made by running the following commands:

   You can rotate the keys for one component at a time, or you can rotate all keys (except the usage metering service account key) at once by setting the component to sakeys.

   gkectl update credentials COMPONENT \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config ADMIN_CLUSTER_CONFIG \
       --admin-cluster
   
   gkectl update credentials COMPONENT \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config USER_CLUSTER_CONFIG
   

   Replace the following;

   • COMPONENT: one of:

     • componentaccess
     • register
     • cloudauditlogging
     • usagemetering
     • stackdriver
     • sakeys (rotates all component keys, except for usage metering)

   • ADMIN_CLUSTER_KUBECONFIG: the path to the kubeconfig file for the admin cluster.

   • ADMIN_CLUSTER_CONFIG: the path to the admin cluster configuration file.

   • USER_CLUSTER_CONFIG: the path to the user cluster configuration file.

Node re-creation

Some service account key rotations may take longer time because node re-creation is required:

For a key rotation that requires nodes to be re-created, the nodes are replaced in a rolling update process; that is, the nodes are re-created one by one.

The possible downtime during a key rotation is similar to the downtime for a cluster upgrade. For details, see Downtime during upgrades.

Restoring backups

If you need to restore the backups of the secrets you made earlier, run the following command:

kubectl apply -f backup/

Advanced cluster differences for rotating service account keys

The gkectl update credentials command is only partially supported for advanced clusters. The following table lists command variations that are and aren't supported:

For a comprehensive list of Google Distributed Cloud for VMware features and capabilities that are or aren't supported for advanced clusters, see Differences when running advanced clusters.