Google Distributed Cloud use certificates and private keys to authenticate
communication between Kubernetes system components in an admin cluster. When you
create an admin cluster, new certificate authority (CA) certificates are created,
and these root certificates are used to issue additional leaf certificates for
Kubernetes system components.
There are three CA certificates used by the Kubernetes system in an admin
cluster:
The etcd CA certificate secures communication from the Kubernetes API server
to the etcd replicas and also communication between etcd replicas. This
certificate is self-signed.
The cluster CA certificate secures communication between the Kubernetes API
server and all internal Kubernetes API clients, for example, the kubelet, the
controller manager, and the scheduler. This certificate is self-signed.
The front-proxy CA certificate secures communication with
aggregated APIs.
This certificate is self-signed.
You can use gkectl to trigger a certificate rotation. During a rotation,
gkectl replaces the core system CA certificates for the
admin cluster with newly generated certificates. Then it distributes the new
CA certificates, leaf certificates, and private keys to admin cluster system
components. The rotation happens incrementally, so that system components can
continue to communicate during the rotation. Note, however, that workloads and
nodes are restarted during the rotation.
Limitations
CA certificate rotation limited to the etcd, cluster, and front-proxy
certificates mentioned previously.
CA certificate rotation is limited to certificates issued automatically by
Google Distributed Cloud. It does not update certificates issued manually by an
administrator, even if those certificates are signed by the system CAs.
CA certificate rotation restarts the Kubernetes API server, other
control-plane processes, and each node in the admin cluster multiple times.
Each stage of a rotation progresses similarly to a cluster upgrade. While the
admin cluster and the user clusters managed by the admin cluster do remain
operational during a certificate rotation, you should expect that workloads
in the admin cluster will be restarted and rescheduled. You should also expect
brief periods of downtime for the admin cluster control plane and user cluster
control plane.
You must update the admin cluster kubeconfig file in the middle of a
certificate rotation and again after the rotation completes. This is because
the old cluster certificate is revoked, and the credentials in the kubeconfig
file will no longer work.
Once initiated, a CA certificate rotation cannot be rolled back.
A CA certificate rotation might take considerable time to complete, depending
on the size of the cluster.
The certificate rotation process can be resumed by re-running the same
command if it is interrupted. However, you must ensure that there is only one
rotation command running at a time.
Start the rotation
To start the certificate rotation, run the following command. It performs the
first half of the rotation and stops at a pause point.
ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration
file
ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig
file
If the command is interrupted, resume it by running the same command.
Update the kubeconfig file
When the preceding command pauses, update the kubeconfig file for the admin
cluster. This places a new client certificate and a new CA certificate in the
kubeconfig file. The old client certificate is removed from the kubeconfig file,
and the old CA certificate remains in the kubeconfig file.
Run the following command to perform the second half of the procedure. The
command will not proceed until gkectl verifies that the updated kubeconfig file
is in the current directory.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eGoogle Distributed Cloud admin clusters use three self-signed CA certificates (etcd, cluster, and front-proxy) to secure communication between Kubernetes system components.\u003c/p\u003e\n"],["\u003cp\u003eCertificate rotation in the admin cluster, initiated with \u003ccode\u003egkectl\u003c/code\u003e, replaces existing CA certificates, leaf certificates, and private keys with newly generated ones, ensuring continued communication among components during the process.\u003c/p\u003e\n"],["\u003cp\u003eDuring CA certificate rotation, the old CA certificates are revoked, requiring users to update the admin cluster kubeconfig file twice: once when the rotation pauses and again upon completion.\u003c/p\u003e\n"],["\u003cp\u003eCA certificate rotation has limitations, such as only applying to the etcd, cluster, and front-proxy certificates issued by Google Distributed Cloud and cannot be rolled back once started.\u003c/p\u003e\n"],["\u003cp\u003eCertificate rotation can be resumed if interrupted, however it can take some time and the admin and user cluster control planes will experience brief periods of downtime.\u003c/p\u003e\n"]]],[],null,["# Rotating admin cluster CA certificates\n\n\u003cbr /\u003e\n\nGoogle Distributed Cloud use certificates and private keys to authenticate\ncommunication between Kubernetes system components in an admin cluster. When you\ncreate an admin cluster, new certificate authority (CA) certificates are created,\nand these root certificates are used to issue additional leaf certificates for\nKubernetes system components.\n\nThis guide applies only to rotation of admin cluster CA certificates. For\nuser clusters, see\n[Rotating user cluster CA certificates](/anthos/clusters/docs/on-prem/1.11/how-to/ca-rotation).\n\nThere are three CA certificates used by the Kubernetes system in an admin\ncluster:\n\n- The etcd CA certificate secures communication from the Kubernetes API server\n to the etcd replicas and also communication between etcd replicas. This\n certificate is self-signed.\n\n- The cluster CA certificate secures communication between the Kubernetes API\n server and all internal Kubernetes API clients, for example, the kubelet, the\n controller manager, and the scheduler. This certificate is self-signed.\n\n- The front-proxy CA certificate secures communication with\n [aggregated APIs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/).\n This certificate is self-signed.\n\nYou can use `gkectl` to trigger a certificate rotation. During a rotation,\n`gkectl` replaces the core system CA certificates for the\nadmin cluster with newly generated certificates. Then it distributes the new\nCA certificates, leaf certificates, and private keys to admin cluster system\ncomponents. The rotation happens incrementally, so that system components can\ncontinue to communicate during the rotation. Note, however, that workloads and\nnodes are restarted during the rotation.\n| **Warning:** A CA certificate rotation revokes the old CA certificates at the end of the operation. This invalidates kubeconfig files that were based on an old certificate. Consequently, the credentials in the kubeconfig file stop working after a CA certificate rotation. This guide includes instructions on how to update your kubeconifg file. The same issue applies to authentication configuration files.\n\nLimitations\n-----------\n\n- CA certificate rotation limited to the etcd, cluster, and front-proxy\n certificates mentioned previously.\n\n- CA certificate rotation is limited to certificates issued automatically by\n Google Distributed Cloud. It does not update certificates issued manually by an\n administrator, even if those certificates are signed by the system CAs.\n\n- CA certificate rotation restarts the Kubernetes API server, other\n control-plane processes, and each node in the admin cluster multiple times.\n Each stage of a rotation progresses similarly to a cluster upgrade. While the\n admin cluster and the user clusters managed by the admin cluster do remain\n operational during a certificate rotation, you should expect that workloads\n in the admin cluster will be restarted and rescheduled. You should also expect\n brief periods of downtime for the admin cluster control plane and user cluster\n control plane.\n\n- You must update the admin cluster kubeconfig file in the middle of a\n certificate rotation and again after the rotation completes. This is because\n the old cluster certificate is revoked, and the credentials in the kubeconfig\n file will no longer work.\n\n- Once initiated, a CA certificate rotation cannot be rolled back.\n\n- A CA certificate rotation might take considerable time to complete, depending\n on the size of the cluster.\n\n- The certificate rotation process can be resumed by re-running the same\n command if it is interrupted. However, you must ensure that there is only one\n rotation command running at a time.\n\nStart the rotation\n------------------\n\nTo start the certificate rotation, run the following command. It performs the\nfirst half of the rotation and stops at a pause point.\n\nStart the rotation:\n\n```\ngkectl update credentials certificate-authorities rotate \\\n --admin-cluster \\\n --config ADMIN_CLUSTER_CONFIG \\\n --kubeconfig ADMIN_CLUSTER_KUBECONFIG\n```\n\nReplace the following:\n\n- \u003cvar translate=\"no\"\u003eADMIN_CLUSTER_CONFIG\u003c/var\u003e: the path of the admin cluster configuration\n file\n\n- \u003cvar translate=\"no\"\u003eADMIN_CLUSTER_KUBECONFIG\u003c/var\u003e: the path of the admin cluster kubeconfig\n file\n\nIf the command is interrupted, resume it by running the same command.\n\nUpdate the kubeconfig file\n--------------------------\n\nWhen the preceding command pauses, update the kubeconfig file for the admin\ncluster. This places a new client certificate and a new CA certificate in the\nkubeconfig file. The old client certificate is removed from the kubeconfig file,\nand the old CA certificate remains in the kubeconfig file.\n\n```\ngkectl update credentials certificate-authorities update-kubeconfig \\\n --admin-cluster \\\n --config ADMIN_CLUSTER_CONFIG \\\n --kubeconfig ADMIN_CLUSTER_KUBECONFIG\n```\n\nContinue the rotation\n---------------------\n\nRun the following command to perform the second half of the procedure. The\ncommand will not proceed until `gkectl` verifies that the updated kubeconfig file\nis in the current directory.\n\n```\ngkectl update credentials certificate-authorities rotate \\\n --admin-cluster \\\n --complete \\\n --config ADMIN_CLUSTER_CONFIG \\\n --kubeconfig ADMIN_CLUSTER_KUBECONFIG\n```\n\nIf the command is interrupted, resume it by running the same command.\n\nWhen the rotation is complete, it reports the current CA version.\n\nUpdate the kubeconfig file again\n--------------------------------\n\nAfter the second half of the rotation completes, update the kubeconfig file\nagain. This removes the old CA certificate from the kubeconfig file.\n\n```\ngkectl update credentials certificate-authorities update-kubeconfig \\\n --admin-cluster \\\n --config ADMIN_CLUSTER_CONFIG \\\n --kubeconfig ADMIN_CLUSTER_KUBECONFIG\n```\n\nDistribute the new kubeconfig file\n----------------------------------\n\nDistribute the new admin cluster kubeconfig file to all cluster users."]]
