About Backup for GKE CMEK encryption

Autopilot Standard

This page describes how customer-managed encryption key (CMEK) support works in Backup for GKE.

Overview

There are two types of user data artifacts that are produced and stored by Backup for GKE:

Config backup: a set of Kubernetes resource descriptions extracted from the API server of the cluster undergoing backup, capturing the cluster state.
Volume backups: a set of volume backups that correspond to PersistentVolumeClaim resources found in the config backup.

By default, all backup artifacts produced by Backup for GKE are encrypted at rest using a Google-supplied key.

However, you may choose to have these artifacts encrypted using a customer-managed encryption key (CMEK) managed with the Cloud Key Management Service.

Enable CMEK encryption

Enabling CMEK encryption involves two steps:

Designate a key for encrypting backups produced for a BackupPlan.
Grant access by the appropriate service accounts to the appropriate keys.

For any particular backup scenario, there are potentially three CMEK keys involved:

bplan_key: This is the key that you reference when creating or updating the BackupPlan. When possible, this key that will be used when encrypting all backup artifacts. This key must live in the same region as the BackupPlan itself (see About resource locations).
orig_disk_key: If you have encrypted your persistent disk volumes using a CMEK key, then the volume backups that Backup for GKE produces for those volumes will be encrypted with this key, even if a different key is registered with the BackupPlan.
new_disk_key: This is the CMEK key you want to use for encrypting volumes you have restored from backup. This is referenced by the StorageClass in the restore's target cluster.

There are five different service accounts that may require access to CMEK keys:

agent_wi: If you are running the Preview version of the agent (GKE clusters running version 1.23 or lower), this service account must be granted access to the bplan_key. This service account will have the form: PROJECT_ID.svc.id.goog[gkebackup/agent], where PROJECT_ID is the ID of your Google Cloud project.
agent_robot: If your GKE clusters are running version 1.24 or higher, this service account must be granted access to the bplan_key. This service account will have the form: service-PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com, where PROJECT_NUMBER is the number of your Google Cloud project.
non_cmek_service_agent: When backing up non-CMEK-encrypted volumes, this service account must be granted access to the bplan_key. This service account will have the form: service-PROJECT_NUMBER@gcp-sa-gkebackup.iam.gserviceaccount.com, where PROJECT_NUMBER is the number of your Google Cloud project.
cmek_service_agent: When backing up CMEK-encrypted volumes, this service account must be granted access to the orig_disk_key. This service account will have the form: service-TENANT_PROJECT_NUMBER@compute-system.iam.gserviceaccount.com, where TENANT_PROJECT_NUMBER is the number of the tenant project assigned for your BackupPlan.

Caution: If the backup scope includes a CMEK-encrypted volume, the VolumeBackup associated with that volume will fail. The stateMessage field of the VolumeBackup resource will contain an error message that identifies the service account (the cmek_service_agent) that must be granted access to orig_disk_key. You can view VolumeBackup resources using the gcloud beta container backup-restore volume-backups list command, passing in --format=yaml to see all fields.
compute_service_agent: This service account is used when creating new encrypted volumes for a cluster and must be granted access to the new_disk_key. This service account will have the form: service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com, where PROJECT_NUMBER is the number of your Google Cloud project.

If diskEncryptionKey.kmsKeyServiceAccount is set for the disks, you will need to perform the following steps before creating a backup:

Disable the organization policy iam.disableCrossProjectServiceAccountUsage to enable service account impersonation across projects:

  gcloud resource-manager org-policies disable-enforce \
      iam.disableCrossProjectServiceAccountUsage
      --project=PROJECT_ID

Grant cmek_service_agent the roles/iam.serviceAccountTokenCreator role to create short-lived credentials:

  gcloud iam service-accounts add-iam-policy-binding \
    # Replace the email with the value from
    # `diskEncryptionKey.kmsKeyServiceAccount`
    your-kms-key-service-acount@PROJECT_ID.iam.gserviceaccount.com \
    --member=service-TENANT_PROJECT_NUMBER@compute-system.iam.gserviceaccount.com \
    --role=roles/iam.serviceAccountTokenCreator

The following table summarizes which service accounts must be granted access to which keys in various scenarios:

Artifact	Service account	Key
config backup, 1.23- cluster	agent_wi	bplan_key
config backup, 1.24+ cluster	agent_robot	bplan_key
backup of CMEK-encrypted volume	cmek_service_agent	orig_disk_key
backup of google-encrypted volume	non_cmek_service_agent	bplan_key
new CMEK-encrypted volume created during restore	compute_service_agent	new_disk_key

You can choose to grant access to keys at a project level, which grants access to all keys in that project, or to the individual key.

Grant project level access

You can grant access to keys at a project level, which grants access to all keys in that project.

Use the following instructions to grant access at a project level.

Console

In the Google Cloud console, go to the IAM page.

Go to IAM
Click Grant access.
In the New principals field, enter the email address of the service account.
From the Select a role list, select the Cloud KMS CryptoKey Encrypter/Decrypter role.
Click Save.

gcloud

Grant access at a project level.

gcloud projects add-iam-policy-binding PROJECT_ID \
--member serviceAccount:service-PROJECT_NUMBER@gcp-sa-gkebackup.iam.gserviceaccount.com \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter

Replace the following:

PROJECT_ID: the ID of the project you want to grant access to.
PROJECT_NUMBER: the number of the project you want to grant access to.

Terraform

Grant access at a project level.

resource "google_project_iam_member" "example_iam_member" {
project = "PROJECT_ID"
role    = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
member  = "serviceAccount:service-PROJECT_NUMBER@gcp-sa-gkebackup.iam.gserviceaccount.com"
}

Replace the following:

PROJECT_ID: the ID of the project you want to grant access to.
PROJECT_NUMBER: the number of the project you want to grant access to.

Grant access at key level

Use the following instructions to grant access at the individual key level.

Console

In the Google Cloud console, go to the Key management page.

Go to Key management
Click the name of the key ring.
Click the name of the key.
Click the Permissions tab.
Click Grant access.
In the New principals field, enter the email address of the service account.
From the Select a role list, select the Cloud KMS CryptoKey Encrypter/Decrypter role.
Click Save.

gcloud

Grant access at the individual key level.
```
gcloud kms keys add-iam-policy-binding KEY_NAME \
--keyring KEY_RING \
--location LOCATION \
--member "serviceAccount:service-PROJECT_NUMBER@gcp-sa-gkebackup.iam.gserviceaccount.com" \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter
```
Replace the following:
- KEY_NAME: the name of the key.
- KEY_RING: the name of the key ring that contains the key.
- LOCATION: the Cloud KMS location of the key ring.
- PROJECT_NUMBER: the number of the project you want to grant access to.

Terraform

Grant access at the individual key level.
```
resource "google_kms_crypto_key_iam_member" "crypto_key_iam_member" {
crypto_key_id = "projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME"
role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
member        = "serviceAccount:service-PROJECT_NUMBER@gcp-sa-gkebackup.iam.gserviceaccount.com" 
}
```
Replace the following:
- PROJECT_ID: the ID of the project you want to grant access to.
- LOCATION: the Cloud KMS location of the key ring.
- KEY_RING: the name of the key ring that contains the key.
- KEY_NAME: the name of the key.
- PROJECT_NUMBER: the number of the project you want to grant access to.

Usage considerations and limitations

If you want to back up a CMEK-encrypted volume, you must grant access to that disk's key, even if you don't enable CMEK encryption in your BackupPlan.
CMEK keys must be in the same region as the BackupPlan to ensure that a regional outage won't remove access to the key while backups are still accessible. However, this constraint cannot be enforced for keys shared with encrypted volumes. When encrypted volumes are involved, it is possible that a restoration may fail even when a backup is available, because the disk encryption key may not be stored in the same region as the backup.