Deploy GPU workloads in Autopilot

This page shows you how to request GPUs to accelerate tasks in your Google Kubernetes Engine (GKE) Autopilot workloads. This page also describes how Autopilot runs GPUs, how your pricing model changes depending on your GKE version, how to set Pod resource requests and limits, and how to monitor GPU workloads.

This page is for Platform admins and operators and for Data and AI specialists who want to request GPUs for workloads that run tasks like machine learning (ML) training or inference. To learn more about the common roles, responsibilities, and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

Before you proceed, ensure that you're familiar with the following concepts:

• About GPUs in GKE
• Kubernetes node selectors

About selecting accelerators in Pods

Autopilot uses the specialized Accelerator compute class to run GPU Pods. With this compute class, GKE places Pods on GPU nodes, providing the Pods with access to advanced capabilities on the virtual machine (VM). To use this class in a GPU workload, take one of the following actions depending on your GKE version:

• Version 1.29.4-gke.1427000 and later: Request GPUs in your workload manifest. You can also use GPU sharing capabilities, like time-sharing. GKE doesn't modify your workload manifests to add a node selector or annotation for the Accelerator class.
• Version 1.29 up to, but not including, version 1.29.4-gke.142700: Specify the cloud.google.com/compute-class: Accelerator node selector in your Pod manifest and request GPUs. If you specify this node selector, you can also use GPU sharing capabilities, like time-sharing.
• Version 1.28.9-gke.1069000 up to, but not including, version 1.29: Specify the cloud.google.com/compute-class: Accelerator node selector in your Pod manifest alongside the GPU selectors. If you specify this node selector, you can also use GPU sharing capabilities, like time-sharing.

The Accelerator compute class isn't supported in versions earlier than 1.28.9-gke.1069000. Instead, GKE treats GPU Pods on those versions similarly to other Autopilot Pods, and you're billed for the resource requests. For details, see Pricing.

Compatibility with GKE capabilities

The following table shows the compatible GKE capabilities for each method of selecting accelerators in GKE Autopilot:

Pricing

The following table describes how the billing model that GKE uses depends on the GKE version of your cluster. For a description of the GKE Autopilot billing models, see Autopilot pricing.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

• Ensure that you have a GKE Autopilot cluster running one of the following versions:

  • Accelerator compute class: Any patch version of 1.28 starting with 1.28.6-gke.1095000
    • NVIDIA H100 Mega (80GB) GPUs: 1.28.9-gke.1250000 or later, and 1.29.4-gke.1542000 or later
    • NVIDIA H100 (80GB) GPUs: 1.28.6-gke.1369000 or later, and 1.29.1-gke.1575000 or later
    • Multiple GPU Pods per VM: 1.29.2-gke.1355000 or later

  • No compute class selection:

    • NVIDIA L4 GPUs: 1.28.3-gke.1203000 or later
    • NVIDIA A100 (80GB) GPUs: 1.27 or later

• Ensure that you have enough GPU quotas available in your project. You must have enough Compute Engine GPU quota for the GPU models that you want to create in each region. If you require additional GPU quota, request GPU quota.

Limitations

• Time-sharing GPUs and multi-instance GPUs are available with Autopilot on GKE version 1.29.3-gke.1093000 and later.
• GPU availability depends on the Google Cloud region of your Autopilot cluster, and your GPU quota. To find a GPU model by region or zone, see GPU regions and zones availability.
• For NVIDIA A100 (80GB) GPUs, you're charged a fixed price for the Local SSDs attached to the nodes, regardless of whether your Pods use that capacity.
• For GKE versions prior to 1.29.2-gke.1355000, if you explicitly request a specific existing GPU node for your Pod, the Pod must consume all the GPU resources on the node. For example, if the existing node has 8 GPUs and your Pod's containers request a total of 4 GPUs, Autopilot rejects the Pod.
• For GKE version 1.29.2-gke.1355000 or later, if you want multiple GPU pods to fit into a single node, the sum of GPU requests for those pods must be less than or equal to the number of GPU resources attached to that node. For example, a node with a gke-accelerator-count of 4 could accommodate up to four Pods that request one GPU each.

Placing multiple Pods on a single GPU node is useful in situations like the following:

• You have capacity reservations for large Accelerator machine types and you run single-GPU workloads, so deploying one Pod per node would waste the other GPUs on that machine
• You have GPU workloads that must run on the same host

In these situations, we recommend that you use all of the GPUs on the node by ensuring that the sum of Pod GPU resource requests on the node is equal to the number of GPUs attached to the node.

Request GPUs in your containers

To request GPU resources for your containers, add the following fields to your Pod specification. Depending on your workload requirements, you can optionally omit the cloud.google.com/gke-accelerator-count selector.

apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-pod
spec:
  nodeSelector:
    cloud.google.com/gke-accelerator: GPU_TYPE
    cloud.google.com/gke-accelerator-count: GPU_COUNT
  containers:
  - name: my-gpu-container
    image: nvidia/cuda:11.0.3-runtime-ubuntu20.04
    command: ["/bin/bash", "-c", "--"]
    args: ["while true; do sleep 600; done;"]
    resources:
      limits:
        nvidia.com/gpu: GPU_QUANTITY

Replace the following:

• GPU_TYPE: the type of GPU hardware. Allowed values are the following:
  • nvidia-h100-mega-80gb: NVIDIA H100 Mega (80GB)
  • nvidia-h100-80gb: NVIDIA H100 (80GB)
  • nvidia-a100-80gb: NVIDIA A100 (80GB)
  • nvidia-tesla-a100: NVIDIA A100 (40GB)
  • nvidia-l4: NVIDIA L4
  • nvidia-tesla-t4: NVIDIA T4
• GPU_COUNT: the total number of GPUs available to attach to the node. Must be greater than or equal to GPU_QUANTITY and a supported GPU quantity for the GPU type you selected. If you omit this nodeSelector, Autopilot places one Pod on each GPU node.
• GPU_QUANTITY: the number of GPUs to allocate to the container. Must be less than or equal to GPU_COUNT and a supported GPU quantity for the GPU type you selected.

For details about how you're billed for accelerator usage in Autopilot mode, see the Pricing section.

You must specify both the GPU type and the GPU quantity in your Pod specification. If you omit either of these values, Autopilot rejects your Pod.

When you deploy this manifest, Autopilot automatically installs the default NVIDIA drivers for the node GKE version. In version 1.29.2-gke.1108000 and later, you can optionally choose to install the latest driver version for that GKE version by adding the following node selector to your manifest:

spec:
  nodeSelector:
    cloud.google.com/gke-gpu-driver-version: "DRIVER_VERSION"

Replace DRIVER_VERSION with one of the following values:

• default - the default, stable driver for your node GKE version. If you omit the nodeSelector in your manifest, this is the default option.
• latest - the latest available driver version for your node GKE version.

CPU and memory requests for Autopilot GPU Pods

When defining your GPU Pods, you should also request CPU and memory resources so that your containers perform as expected. Autopilot enforces specific CPU and memory minimums, maximums, and defaults based on the GPU type and quantity. If you run multiple GPU Pods on a single node, specify the CPU and memory, otherwise it defaults to the node's entire capacity. For details, refer to Resource requests in Autopilot.

Your Pod specification should look similar to the following example, which requests four T4 GPUs:

apiVersion: v1
kind: Pod
metadata:
  name: t4-pod
spec:
  nodeSelector:
    cloud.google.com/gke-accelerator: "nvidia-tesla-t4"
  containers:
  - name: t4-container-1
    image: nvidia/cuda:11.0.3-runtime-ubuntu20.04
    command: ["/bin/bash", "-c", "--"]
    args: ["while true; do sleep 600; done;"]
    resources:
      limits:
        nvidia.com/gpu: 3
        cpu: "54"
        memory: "54Gi"
      requests:
        cpu: "54"
        memory: "54Gi"
  - name: t4-container-2
    image: nvidia/cuda:11.0.3-runtime-ubuntu20.04
    command: ["/bin/bash", "-c", "--"]
    args: ["while true; do sleep 600; done;"]
    resources:
      limits:
        nvidia.com/gpu: 1
        cpu: "18"
        memory: "18Gi"
      requests:
        cpu: "18"
        memory: "18Gi"

This manifest specifies limits for CPU and memory resources. If you omit the limits for CPU or memory in GKE version 1.29.2-gke.1060000 and later, GKE gives your Pods the Burstable QoS class and lets your Pods burst into unused resources from the sum of resource requests on the node. For more information, see Configure Pod bursting in GKE.

Ephemeral storage requests for Autopilot GPU Pods

You can also request ephemeral storage in Pods that need short-lived storage. The maximum available ephemeral storage and the type of storage hardware used depends on the type and quantity of GPUs the Pod requests. You can use Local SSD for ephemeral storage if using NVIDIA L4 GPUs, the Accelerator compute class, and running GKE patch version 1.28.6-gke.1369000 and later or 1.29.1-gke.1575000 and later.

To use Local SSD for ephemeral storage, add the cloud.google.com/gke-ephemeral-storage-local-ssd: "true" nodeSelector to your workload manifest. See the example manifest in Use Local SSD-backed ephemeral storage with Autopilot clusters. The NVIDIA H100 (80GB) GPUs and NVIDIA A100 (80GB) GPUs always use Local SSDs for ephemeral storage, and you can't specify this node selector for those GPUs.

Verify GPU allocation

To check that a deployed GPU workload has the requested GPUs, run the following command:

kubectl describe node NODE_NAME

Replace NODE_NAME with the name of the node on which the Pod was scheduled.

The output is similar to the following:

apiVersion: v1
kind: Node
metadata:
...
  labels:
    ...
    cloud.google.com/gke-accelerator: nvidia-tesla-t4
    cloud.google.com/gke-accelerator-count: "1"
    cloud.google.com/machine-family: custom-48
    ...
...

Check GPU driver version

In Autopilot clusters, GKE automatically installs NVIDIA device drivers on all GPU nodes. To find the driver version that GKE installed in your cluster, run the following command:

kubectl logs --selector=k8s-app=nvidia-gpu-device-plugin \
    --container="nvidia-gpu-device-plugin" \
    --tail=-1 \
    --namespace=kube-system | grep Driver

The output is similar to the following:

I1206 18:37:08.251742    5851 metrics.go:144] nvml initialized successfully. Driver version: 535.104.12

How GPU allocation works in Autopilot

After you request a GPU type and a quantity for the containers in a Pod and deploy the Pod, the following happens:

1. If no allocatable GPU node exists, Autopilot provisions a new GPU node to schedule the Pod. Autopilot automatically installs NVIDIA's drivers to facilitate the hardware.
2. Autopilot adds node taints to the GPU node and adds the corresponding tolerations to the Pod. This prevents GKE from scheduling other Pods on the GPU node.

Autopilot places exactly one GPU Pod on each GPU node, as well as any GKE-managed workloads that run on all nodes, and any DaemonSets that you configure to tolerate all node taints.

Run DaemonSets on every node

You might want to run DaemonSets on every node, even nodes with applied taints. For example, some logging and monitoring agents must run on every node in the cluster. You can configure those DaemonSets to ignore node taints so that GKE places those workloads on every node.

To run DaemonSets on every node in your cluster, including your GPU nodes, add the following toleration to your specification:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logging-agent
spec:
  tolerations:
  - key: ""
    operator: "Exists"
    effect: ""
  containers:
  - name: logging-agent-v1
    image: IMAGE_PATH

To run DaemonSets on specific GPU nodes in your cluster, add the following to your specification:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logging-agent
spec:
  nodeSelector:
    cloud.google.com/gke-accelerator: "GPU_TYPE"
  tolerations:
  - key: ""
    operator: "Exists"
    effect: ""
  containers:
  - name: logging-agent-v1
    image: IMAGE_PATH

Replace GPU_TYPE with the type of GPU in your target nodes. This can be one of the following:

• nvidia-h100-mega-80gb: NVIDIA H100 Mega (80GB)
• nvidia-h100-80gb: NVIDIA H100 (80GB)
• nvidia-a100-80gb: NVIDIA A100 (80GB)
• nvidia-tesla-a100: NVIDIA A100 (40GB)
• nvidia-l4: NVIDIA L4
• nvidia-tesla-t4: NVIDIA T4

GPU use cases in Autopilot

You can allocate GPUs to containers in Autopilot Pods to facilitate workloads such as the following:

• Machine learning (ML) inference
• ML training
• Rendering

Supported GPU quantities

When you request GPUs in your Pod specification, you must use the following quantities based on the GPU type:

If you request a GPU quantity that isn't supported for that type, Autopilot rejects your Pod.

Monitor your GPU node workload performance

If your GKE cluster has system metrics enabled, then the following metrics are available in Cloud Monitoring to monitor your GPU workload performance:

• Duty Cycle (container/accelerator/duty_cycle): Percentage of time over the past sample period (10 seconds) during which the accelerator was actively processing. Between 1 and 100.
• Memory Usage (container/accelerator/memory_used): Amount of accelerator memory allocated in bytes.
• Memory Capacity (container/accelerator/memory_total): Total accelerator memory in bytes.

You can use predefined dashboards to monitor your clusters with GPU nodes. For more information, see View observability metrics. For general information about monitoring your clusters and their resources, refer to Observability for GKE.

View usage metrics for workloads

You view your workload GPU usage metrics from the Workloads dashboard in the Google Cloud console.

To view your workload GPU usage, perform the following steps:

1. Go to the Workloads page in the Google Cloud console.

   Go to Workloads
2. Select a workload.

The Workloads dashboard displays charts for GPU memory usage and capacity, and GPU duty cycle.

View NVIDIA Data Center GPU Manager (DCGM) metrics

You can collect and visualize NVIDIA DCGM metrics by using Google Cloud Managed Service for Prometheus. For Autopilot clusters, GKE installs the drivers. For Standard clusters, you must install the NVIDIA drivers.

For instructions on how to deploy the GKE-managed DCGM package, see Collect and view NVIDIA Data Center GPU Manager (DCGM) metrics.

What's next

• Learn more about GPU support in GKE.
• Read about how Autopilot compute classes are optimized for specialized use cases.
• Read about deploying GPUs for batch workloads with Dynamic Workload Scheduler.