About GPUs in Google Kubernetes Engine (GKE)

This page describes GPUs in Google Kubernetes Engine (GKE) to help you to select the optimal GPU configuration for your workloads. You can use GPUs to accelerate resource intensive tasks, such as machine learning and data processing. The information on this page can help you to do the following:

• Ensure GPU availability when needed.
• Decide whether to use GPUs in GKE Autopilot mode or GKE Standard mode clusters.
• Choose GPU-related features to efficiently use your GPU capacity.
• Monitor GPU node metrics.
• Improve GPU workload reliability by handling disruptions more effectively.

This page is intended for Platform admins and operators and Machine learning (ML) engineers who want to ensure that accelerator infrastructure is optimized for your workloads.

Before reading this page, ensure that you're familiar with the following:

• Autopilot mode and Standard mode
• Quotas in Google Cloud

GPU selection in GKE

In GKE, the way you request GPU hardware depends on whether you are using Autopilot or Standard mode. In Autopilot, you request GPU hardware by specifying GPU resources in your workloads. In GKE Standard mode, you can attach GPU hardware to nodes in your clusters, and then allocate GPU resources to containerized workloads running on those nodes. For detailed instructions on how to attach and use GPUs in your workloads, refer to Deploy GPU workloads on Autopilot or Run GPUs on Standard node pools.

The GPU hardware that's available for use in GKE is a subset of the Compute Engine GPUs for compute workloads. The specific hardware that's available depends on the Compute Engine region or zone of your cluster. For specific availability, refer to GPU regions and zones.

GKE offers some GPU-specific features to improve efficient GPU resource utilization of workloads running on your nodes, including time-sharing, multi-instance GPUs, and multi-instance GPUs with NVIDIA MPS.

This page helps you to consider choices for requesting GPUs in GKE, including the following:

• Choosing your GPU quota, the maximum number of GPUs that can run in your project
• Deciding between Autopilot and Standard modes
• Manage the GPU stack through GKE or NVIDIA GPU Operator on GKE
• Choosing features to reduce the amount of underutilized GPU resources
• Accessing NVIDIA CUDA-X libraries for CUDA applications
• Monitoring GPU node metrics
• Handling disruption due to node maintenance

Plan GPU quota

Your GPU quota is the maximum number of GPUs that can run in your Google Cloud project. To use GPUs in your GKE clusters, your project must have enough GPU quota.

Your GPU quota should be at least equal to the total number of GPUs you intend to run in your cluster. If you enable cluster autoscaling, you should request GPU quota at least equivalent to your cluster's maximum number of nodes multiplied by the number of GPUs per node.

For example, if you expect to utilize three nodes with two GPUs each, then six is the GPU quota required for your project.

To request additional GPU quota, follow the instructions in Requesting a higher quota limit, using gpus as the metric.

Choose GPU support using Autopilot or Standard

GPUs are available in Autopilot and Standard clusters.

The following table provides an overview of the differences between Autopilot and Standard GPU support:

To choose the GKE mode of operation that's the best fit for your workloads, see Choose a GKE mode of operation.

Manage the GPU stack through GKE or the NVIDIA GPU Operator on GKE

By default, GKE manages the entire lifecycle of the GPU nodes, including automatic GPU driver installation, monitoring GPU workloads on GKE with NVIDIA Data Center GPU Manager (DCGM), and GPU sharing strategies.

Get started with GKE for GPU node management by choosing one of the following:

• Deploy GPU workloads in Autopilot
• Run GPUs in Standard node pools

The NVIDIA GPU Operator may be used as an alternative to fully managed GPU support on GKE on both Container-Optimized OS (COS) and Ubuntu node images. Select this option if you are looking for a consistent experience across multiple cloud service providers, you are already using the NVIDIA GPU Operator, or if you are using software that depends on the NVIDIA GPU operator. To learn more, see Manage the GPU stack with the NVIDIA GPU Operator.

To select the best option for your use case, refer to the following table comparing the two methods of managing GPU nodes on GKE.