This page shows you how to enforce encryption of data in-use in your nodes and workloads using Confidential Google Kubernetes Engine Nodes.
Overview
You can encrypt your workloads with Confidential GKE Nodes or Confidential mode for Hyperdisk Balanced.
Confidential GKE Nodes
Confidential GKE Nodes is built on top of Compute Engine Confidential VM using AMD Secure Encryption Virtualization (SEV), which encrypts the memory contents of VMs in-use. Encryption-in-use is one of the three states of end-to-end encryption.
When you enable Confidential GKE Nodes on a cluster or on a node pool, data in workloads running on the confidential nodes is encrypted-in-use. For visibility over your control plane, use Access Transparency.
You can enable Confidential GKE Nodes when doing one of the following:
- Create a new cluster
- Create a new node pool
- Update an existing node pool
You cannot update an existing cluster to change the cluster-level Confidential GKE Nodes setting.
The following table shows you the GKE behavior that applies when you enable Confidential GKE Nodes at the cluster level or at the node pool level:
Confidential GKE Nodes setting | How to configure | Behavior |
---|---|---|
Cluster-level | Create a new cluster | All nodes in the cluster in any node pool use
Confidential GKE Nodes. You cannot do the
following:
|
Node pool level |
|
You can only configure Confidential GKE Nodes for node pools when this feature is disabled at the cluster-level. |
Confidential mode for Hyperdisk Balanced
You can also enable Confidential mode for Hyperdisk Balanced on your boot disk storage which encrypts your data on additional hardware-backed enclaves.
You can enable Confidential mode for Hyperdisk Balanced when doing one of the following:
- Create a new cluster
- Create a new node pool
You cannot update an existing cluster or a node pool to change the Confidential mode for Hyperdisk Balanced setting.
The following table shows you the GKE behavior that applies when you enable Confidential mode for Hyperdisk Balanced setting at the cluster level or at the node pool level:
Confidential mode for Hyperdisk Balanced setting | How to configure | Behavior |
---|---|---|
Cluster-level | Create a new cluster | Only the default node pool in the cluster will use
Confidential mode for Hyperdisk Balanced setting. You cannot do the
following:
|
Node pool level | Create a new node pool | You can configure Confidential mode for Hyperdisk Balanced setting for any new node pools at creation time. You can't update existing node pools to use Confidential mode for Hyperdisk Balanced setting. |
Pricing
There is no additional cost to deploy Confidential GKE Nodes, other than the cost of Compute Engine Confidential VM. However, Confidential GKE Nodes might generate slightly more log data on startup than standard nodes. For information on logs pricing, see Pricing for Google Cloud Observability.
Availability
Confidential GKE Nodes is available in the following situations:
Confidential GKE Nodes is only available in zones and regions with N2D instances, C2D instances, or C3D instances available.
Confidential GKE Nodes can be used with Container-Optimized OS and Container-Optimized OS with containerd (
cos_containerd
).
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Enable Confidential GKE Nodes on clusters
You can create a new cluster with Confidential GKE Nodes enabled by using the gcloud CLI or the Google Cloud console. If you enable Confidential GKE Nodes at the cluster level, all the nodes in the cluster are Confidential VM.
gcloud
When creating a new cluster, specify the --enable-confidential-nodes
option
in the gcloud CLI:
gcloud container clusters create CLUSTER_NAME \
--machine-type=MACHINE_TYPE \
--enable-confidential-nodes
Replace the following:
Console
Go to the Google Kubernetes Engine page in the Google Cloud console.
Click add_box Create.
In the Standard section, click Configure.
From the navigation pane, under Cluster, click Security.
Select the Enable Confidential GKE Nodes checkbox.
Configure your cluster as needed.
Click Create.
See Creating a regional cluster for more details about creating clusters.
After creating a cluster with Confidential GKE Nodes, any node pools created in this cluster can only use confidential nodes. You cannot create regular node pools in clusters with Confidential GKE Nodes enabled. You also cannot disable Confidential GKE Nodes on individual node pools when you enable Confidential GKE Nodes at the cluster level.
For any node pool created with the Confidential mode for Hyperdisk Balanced setting, only the nodes in the node pool are restricted to the setup configuration. For any new node pools created in the cluster, you must set up confidential mode at creation.
Enable Confidential GKE Nodes on node pools
You can enable Confidential GKE Nodes on specific node pools if Confidential GKE Nodes is disabled at the cluster level.
Confidential mode for Hyperdisk Balanced setting must be specified during node pool creation request.
Create a new node pool
To create a new node pool with Confidential GKE Nodes enabled, run the following command:
gcloud container node-pools create NODE_POOL_NAME \
--cluster=CLUSTER_NAME \
--machine-type=MACHINE_TYPE \
--enable-confidential-nodes
Replace the following:
- NODE_POOL_NAME: the name of your new node pool.
- CLUSTER_NAME: the name of your cluster.
- MACHINE_TYPE: the machine type for your node pool, which must be a N2D, C2D, or C3D machine type.
Update an existing node pool
You can enable Confidential GKE Nodes on existing node pools that use the N2D, C2D, or C3D machine types. Run the following command:
gcloud container node-pools update NODE_POOL_NAME \
--cluster=CLUSTER_NAME \
--enable-confidential-nodes
Replace the following:
- NODE_POOL_NAME: the name of your node pool.
- CLUSTER_NAME: the name of your cluster.
This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy without respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.
Verify that Confidential GKE Nodes are enabled
On clusters
You can verify that your cluster is using Confidential GKE Nodes with the gcloud CLI or the Google Cloud console.
gcloud
Describe the cluster:
gcloud container clusters describe CLUSTER_NAME
If Confidential GKE Nodes is enabled, the output of the command includes the following lines:
confidentialNodes:
enabled: true
Console
Go to the Google Kubernetes Engine page in the Google Cloud console.
Click the name of the cluster you want to inspect.
Under Security, in the Confidential GKE Nodes field, verify that Confidential GKE Nodes is Enabled.
On node pools
To verify that your node pool is using Confidential GKE Nodes, run the following command:
gcloud container node-pools describe NODE_POOL_NAME \
--cluster=CLUSTER_NAME
If Confidential GKE Nodes is enabled, the output is similar to the following:
confidentialNodes:
enabled: true
If Confidential mode for Hyperdisk Balanced setting is enabled, the output is similar to the following:
enableConfidentialStorage: true
On nodes
To validate the confidentiality of specific nodes, you can:
Run applications on Confidential GKE Nodes
Google's approach to Confidential Computing is to enable an effortless lift and shift for existing applications. GKE workloads that you run today can run on Confidential GKE Nodes without code changes.
Optionally, if you want to declaratively express that your workloads must only
run on clusters with Confidential GKE Nodes, you can use the
cloud.google.com/gke-confidential-nodes
node selector.
Here's an example Pod spec that uses this selector:
apiVersion: v1
kind: Pod
spec:
containers:
- name: my-confidential-app
image: us-docker.pkg.dev/myproject/myrepo/my-confidential-app
nodeSelector:
cloud.google.com/gke-confidential-nodes: "true"
Set organization policy constraints
You can define an organization policy constraint to ensure that all VM resources
created across your organization are Confidential VM instances.
For GKE, you can customize the Restrict Non-Confidential
Computing constraint to require that all new clusters are created with
Confidential GKE Nodes enabled. Add the container.googleapis.com
API Service name to the deny list when enforcing organization policy constraints,
for example:
gcloud resource-manager org-policies deny \
constraints/compute.restrictNonConfidentialComputing compute.googleapis.com container.googleapis.com \
--project=PROJECT_ID
Replace PROJECT_ID with your project ID.
Create a PersistentVolume for Confidential mode for Hyperdisk Balanced
For guidance on allowable values for throughput or IOPS, see Plan the performance level for your Hyperdisk volume.
The following examples show how you can create a Confidential mode for Hyperdisk Balanced StorageClass for each Hyperdisk type:
Hyperdisk Balanced
Save the following manifest in a file named
confidential-hdb-example-class.yaml
:apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: balanced-storage provisioner: pd.csi.storage.gke.io volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: hyperdisk-balanced provisioned-throughput-on-create: "250Mi" provisioned-iops-on-create: "7000" enable-confidential-storage: true disk-encryption-kms-key: "projects/KMS_PROJECT_ID/locations/REGION/keyRings/KEY_RING/cryptoKeys/HSM_KEY_NAME"
Replace the following:
KMS_PROJECT_ID
: the project that owns the Cloud KMS keyREGION
: the region where the disk is locatedKEY_RING
: the name of the key ring that includes the keyHSM_KEY_NAME
: the name of the HSM key used to encrypt the disk
Create the StorageClass:
kubectl create -f hdb-example-class.yaml
Create a Hyperdisk Persistent Volume Claim for GKE that uses your Confidential mode for Hyperdisk Balanced volume.
To find the name of the StorageClasses available in your cluster, run the following command:
kubectl get sc
Limitations
Confidential GKE Nodes has the following limitations:
- All the limitations of Compute Engine Confidential VM instances apply to Confidential GKE Nodes.
- Confidential GKE Nodes that have the C2D machine type can only use node auto-provisioning in GKE version 1.24 or later.
- Confidential GKE Nodes only supports PersistentVolumes backed by persistent disks if your control plane runs GKE version 1.22 or later. For instructions, refer to Using the Compute Engine persistent disk CSI Driver .
- Confidential GKE Nodes is not compatible with GPUs.
- Confidential GKE Nodes is not compatible with sole tenant nodes.
- Confidential GKE Nodes only supports using ephemeral storage on local SSDs, but doesn't support using local SSDs in general.
- Only Container-Optimized OS nodes are supported. Ubuntu and Windows nodes are not supported.
- Autopilot clusters don't support Confidential GKE Nodes.
- See Limitations section for additional Confidential mode for Hyperdisk Balanced information.
- Confidential mode for Hyperdisk Balanced is supported only on Confidential GKE Nodes.
Live migration limitations
Certain Compute Engine Confidential VM machine types support live migration, which minimizes the potential workload disruption from a host maintenance event. Live migration occurs in the following GKE versions:
- 1.27.10-gke.1218000 and later
- 1.28.6-gke.1393000 and later
- 1.29.1-gke.1621000 and later
If your node pools were already running a supported version when live migration was added, manually upgrade the node pools to the same or a different supported version. Upgrading the nodes triggers node recreation, and the new nodes have live migration enabled.
For details about which Compute Engine machine types support live migration, see Supported configurations.
If a host maintenance event
occurs on a node that doesn't support live migration, the node enters a
NotReady
state. Running Pods will experience disruptions until the node
becomes ready again. If the maintenance takes more than five minutes,
GKE might try to recreate the Pods on other nodes.
Disable Confidential GKE Nodes
Disabling Confidential GKE Nodes only works for node pools that have enabled Confidential GKE Nodes. If the cluster is created with Confidential GKE Nodes, you cannot disable the feature. Run the following command to disable Confidential GKE Nodes on a node pool:
gcloud container node-pools update NODE_POOL_NAME \
--cluster=CLUSTER_NAME \
--no-enable-confidential-nodes
This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy without respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.
What's next
- Learn more about Confidential VM.
- Learn more about node images.
- Learn more about Google Cloud encryption at rest.
- Learn more about Google Cloud encryption in transit.
- Learn more about customer-managed encryption keys (CMEK).
- Learn more about application-layer secrets encryption.