Rotate etcd and control plane boot disk encryption keys

This page shows you how to rotate the etcd and control plane boot disk encryption keys that you configured for GKE control plane authority. This page is intended for cluster administrators and Security engineers.

Before you read this page, familiarize yourself with the following GKE control plane authority concepts:

Plan key rotations

This page shows you how to rotate the following credential components in your control plane:

  • The encryption key for etcd and the control plane boot disks.
  • The encryption key for the etcd internal backup that Google Cloud uses for disaster recovery.

You can also rotate the customer-managed certificate authorities (CAs) and service account signing keys that you configure with GKE control plane authority. For more information, see Rotate customer-managed control plane CAs and signing keys.

Keys in Cloud Key Management Service (Cloud KMS) don't expire. If your organization has a security requirement for key rotation, manually rotate the keys. To minimize disruptions to running workloads, don't configure automatic key rotation for these keys.

When you rotate the encryption keys for your control plane boot disks and etcd, GKE creates a new disk that's encrypted using the new key. For regional clusters, this operation has no downtime because of the replicated control plane. For zonal clusters, the control plane becomes unavailable until the new disk is available.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

  • Have an existing cluster that uses control plane boot disk and etcd encryption

  • Identify the project IDs of the following Google Cloud projects:

    • Key project: the project that contains your Cloud KMS resources.
    • Cluster project: the project that contains your GKE cluster.

  • To do the validation tasks on this page, verify that the DATA_READ Data Access audit log is enabled for the Cloud Key Management Service (KMS) API. For more information, see Enable Data Access audit logs.

Required roles and permissions

To get the permissions that you need to rotate your customer-managed CAs and keys, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Limitations

You can update your clusters to new versions of existing keys, but you can't update your clusters to use entirely new keys for these encryption operations.

Grant IAM roles to the GKE service agent

To rotate your encryption keys for boot disks and etcd, grant the Kubernetes Engine KMS Crypto Key User (roles/container.cloudKmsKeyUser) role on the encryption keys in your key project to the GKE service agent in your cluster project:

gcloud kms keys add-iam-policy-binding KCP_DISK_KEY_NAME \
    --keyring=KEYRING_NAME \
    --location=LOCATION \
    --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \
    --role=roles/container.cloudKmsKeyUser \
    --project=KEY_PROJECT_ID

Replace the following:

  • KCP_DISK_KEY_NAME: the name of the encryption key for your control plane boot disks and etcd disks.
  • KEYRING_NAME: the name of the key ring that holds the key.
  • LOCATION: the Google Cloud location of the key ring. This must be the same as your cluster location. For a list of regions, filter for "Region" in the Cloud KMS locations table.
  • CLUSTER_PROJECT_NUMBER: the numerical project number of the cluster project.
  • KEY_PROJECT_ID: the project ID of your key project.

Rotate the key for etcd and control plane boot disks

  1. Create a new primary key version for the boot disk and etcd encryption key:

    gcloud kms keys versions create \
    --key=KCP_DISK_KEY_NAME \
    --keyring=KEYRING_NAME \
    --location=LOCATION \
    --project=KEY_PROJECT_ID \
    --primary

    It might take up to 10 minutes for the changes to propagate in the project.

  2. Find the current patch version of the cluster control plane:

    gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --format='value(currentMasterVersion)'

    Replace the following:

    • CLUSTER_NAME: the name of your GKE cluster.
    • LOCATION: the location of the GKE cluster control plane, such as us-central1.

  3. Upgrade the cluster control plane to the same version that it already uses.

    gcloud container clusters upgrade CLUSTER_NAME \
    --master \
    --location=LOCATION \
    --cluster-version=CONTROL_PLANE_VERSION \
    --project=CLUSTER_PROJECT_ID

    Replace the following:

    • CONTROL_PLANE_VERSION: the version of the cluster control plane from the output of the previous step.
    • CLUSTER_PROJECT_ID: the project ID of your cluster project.

After the control plane upgrade operation completes, the cluster uses the new key version to encrypt your control plane boot disks and any etcd disks.

Rotate the etcd internal backup encryption key

This section shows you how to rotate the encryption key for the etcd internal backup, which Google Cloud uses for disaster recovery purposes. You can configure automatic rotation for this key, but we recommend that you perform manual rotation instead. After you rotate the key, the cluster uses the new key version to encrypt new etcd internal backups.

etcd internal backups use the primary key version and occur at the following intervals:

  • Every 30 minutes for zonal clusters.
  • Every 10 minutes for Autopilot clusters and regional Standard clusters.

After you rotate the key, don't immediately disable or destroy the previous key version. Wait until enough time has passed for GKE to create new etcd internal backups that use the new key version.

  • Create a new primary key version for the etcd internal backup encryption key:

    gcloud kms keys versions create \
    --key=ETCD_BACKUP_KEY_NAME \
    --keyring=KEYRING_NAME \
    --location=LOCATION \
    --project=KEY_PROJECT_ID \
    --primary

    Replace the following:

    • ETCD_BACKUP_KEY_NAME: the name of the etcd internal backup encryption key.
    • KEYRING_NAME: the name of the key ring that holds the key.
    • LOCATION: the Google Cloud location of the key ring. This must be the same as your cluster location. For a list of regions, filter for "Region" in the Cloud KMS locations table.
    • KEY_PROJECT_ID: the project ID of your key project.

What's next