Use nested VMs with GKE Standard clusters

The following page explains how you can create Google Kubernetes Engine (GKE) Standard clusters with nodes that support nested virtualization. Compute Engine VMs use Intel VT-x instructions so that nested VMs can run on a GKE node's underlying VM. Then, you can deploy Pods that use tools such as QEMU to create nested VMs. You can use nested VMs to run specialized workloads such as Android emulators, or workloads that benefit from the VM as an isolation boundary.

Reduced performance implications

With hardware-assisted nested virtualization, workloads running on the nested VMs might experience reduced performance when compared to non-nested virtualization configurations. The amount of performance impact will depend on the specific workload profile, including its I/O and memory usage characteristics.

Additionally, creating nested VMs on the GKE nodes' underlying VMs might affect the performance of other workloads running on those nodes.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Requirements and limitations

Consider the following information before enabling nested virtualization with GKE and allowing your Pods to create nested VMs:

  • You can only configure node pools with nested VMs with a limited number of VM machine series. See the Nested virtualization row of the Machine series comparison to confirm supported machine series.
  • You must use the UBUNTU_CONTAINERD image type, or the COS_CONTAINERDimage type with nodes running version 1.28.4-gke.1083000 or later.
  • You can only use nested virtualization with Standard clusters, not Autopilot clusters.
  • You can't use node auto-provisioning with nested virtualization-enabled node pools.
  • You can't use nested virtualization if the Disable VM nested virtualization constraint is enforced for your organization policy.
  • You must set securityContext.privileged:true for Pods to interact with nested VMs.

Enable nested virtualization

You can enable nested virtualization when you create a cluster—by enabling it for the default node pool—or when you create a node pool. You can't enable or disable the feature for an existing node pool.

The instructions in the following sections create node pools with nodes with the label nested-virtualization=enabled. After GKE creates the nodes, you can schedule Pods that created nested VMs on the node's underlying VM.

Enable nested virtualization with a new Standard cluster

Create a new Standard cluster with nested virtualization enabled for the default node pool. GKE only creates the default node pool with nested virtualization enabled. GKE doesn't automatically enable the feature for all new node pools created for the cluster:

gcloud container clusters create CLUSTER_NAME \
    --enable-nested-virtualization \
    --node-labels=nested-virtualization=enabled \
    --machine-type=MACHINE_TYPE

Replace the following:

Optionally, check that nested virtualization is enabled for a node in the node pool.

You can now schedule Pods that create nested VMs on the underlying VM of the node.

Enable nested virtualization with a new node pool

Create a new node pool with nested virtualization enabled for an existing Standard cluster:

gcloud container node-pools create NODEPOOL_NAME \
    --enable-nested-virtualization \
    --node-labels=nested-virtualization=enabled \
    --cluster=CLUSTER_NAME \
    --machine-type=MACHINE_TYPE

Replace the following:

Optionally, check that nested virtualization is enabled for a node in the node pool.

You can now schedule Pods that create nested VMs on the underlying VM of the node.

Schedule a Pod to a node that supports nested virtualization

To schedule a Pod on a node with nested virtualization enabled, add the following node selector to the specification:

nodeSelector:
  nested-virtualization: enabled

Additionally, for the Pod to connect with a nested VM, you must set the Pod as privileged by configuring securityContext.privileged:true.

What's next