Enable and use NVIDIA GPUS in VMs with VM Runtime on GDC

This document shows you how to enable NVIDIA® GPU support for virtual machines (VMs) that run using VM Runtime on GDC. You learn how to install the NVIDIA drivers on your Google Distributed Cloud nodes, verify that the GPUs are available, and assign GPUs to VMs.

Before you begin

To complete this document, you need access to the following resources:

Access to Google Distributed Cloud version 1.12.0 (anthosBareMetalVersion: 1.12.0) or higher cluster. You can use any cluster type capable of running workloads. If needed, try Google Distributed Cloud on Compute Engine or see the cluster creation overview.
The virtctl client tool installed as a plugin for kubectl. If needed, install the virtctl client tool.

Supported Nvidia GPU cards

Google Distributed Cloud version 1.13 or higher support the following NVIDIA GPUs:

Tesla T4
Tesla P4
Tesla V100 SXM2 32 GB
A100 SXM4 40 GB
A100 PCIe 40 GB
A100 SXM4 80 GB
A100 PCIe 80 GB

Install NVIDIA drivers on nodes

Before your VMs can use the NVIDIA GPUs, you must configure your Google Distributed Cloud nodes to support the GPU devices. To install the NVIDIA drivers on your nodes, complete the following steps on each node in your cluster that includes an NVIDIA GPU. This document uses a supported Ubuntu version for the nodes:

Connect to your Google Distributed Cloud node that you want to configure for GPU support.
Get the kernel version of your node:
```
KERNEL_VERSION="$(uname -r)"
```

Update your Ubuntu node and install the appropriate kernel headers:

sudo apt update && \
apt install -y linux-headers-${KERNEL_VERSION}

Install the build-essential package so that you can compile the Nvidia drivers in a following step:
```
sudo apt install -y build-essential
```
Download the appropriate NVIDIA driver package for your GPU. For a complete list of drivers, see NVIDIA Driver Downloads.

The following example downloads the Linux x86_64 version 470.82.01 driver:
```
wget https://us.download.nvidia.com/tesla/470.82.01/NVIDIA-Linux-x86_64-470.82.01.run
```

Install the NVIDIA driver package. Use the name of the NVIDIA driver package you downloaded in the previous step:

sudo sh NVIDIA-Linux-x86_64-470.82.01.run \
  --accept-license \
  --silent \
  --no-nouveau-check

Load the NVIDIA kernel module:
```
sudo modprobe nvidia
```
Repeat the steps in this section on each node in your cluster that has a NVIDIA GPU.

Enable GPU support in VM Runtime on GDC

After you install the NVIDIA drivers on your Google Distributed Cloud node(s), you enable GPU support in VM Runtime on GDC. Your VMs can then access the GPUs on the nodes.

Each node reboots as part of the following process. Your VMs may be affected by this reboot process. If possible and configured to do so, migratable VMs migrate to other nodes. For more information, see how to configure the eviction policy for VMs during maintenance events.

To enable GPU support in VM Runtime on GDC, complete the following steps.

Edit the VMRuntime custom resource:
```
kubectl edit vmruntime vmruntime
```

Add the enableGPU: true property to the VMRuntime manifest:

apiVersion: vm.cluster.gke.io/v1
kind: VMRuntime
metadata:
  name: vmruntime
spec:
  enabled: true
  enableGPU: true
...

Save and close the VMRuntime custom resource in your editor.

Check the status of the GPU controllers in the vm-system namespace:

kubectl get pods --namespace vm-system  -w

It takes about five minutes for the controllers to be successfully enabled. Wait for the STATUS to show Running for all the GPU controllers. The following example output shows the desired state:

NAME                                          READY  STATUS    RESTARTS     AGE
gpu-controller-controller-manager-gwvcb       2/2    Running   0            10m
kubevirt-gpu-dp-daemonset-2lfkl               1/1    Running   0            10m
kubevm-gpu-driver-daemonset-5fwh6             1/1    Running   0            10m
nvidia-gpu-dp-daemonset-9zq2w                 1/1    Running   0            10m
nvidia-mig-manager-5g7pz                      1/1    Running   0            10m
vm-controller-controller-manager-7b6df6979b   2/2    Running   2 (13m ago)  14m

Verify that the GPUs are available for use when the GPU controllers all report their status as Running:
```
kubectl get gpuallocations --namespace vm-system
```
The following example output shows that the GPUs on the nodes are available for use. Each node in your cluster with GPU support is shown. You allocate them to VMs in the next section:
```
NAME       ALLOCATED   DEVICEMODEL
bm-node1   true        Tesla A100 SXM4 40GB
bm-node2   true        Tesla A100 SXM4 40GB
```

Allocate GPUs for use with VMs

With GPU support configured in your bare metal cluster nodes and in VM Runtime on GDC, allocate the GPUs for use with VMs. By default, GPUs are allocated for use with pods (containers).

Edit the GPUAllocation custom resource for use with VMs. This step assigns the GPUs on the nodes for use with VMs:
```
kubectl edit gpuallocation NODE_NAME --namespace vm-system
```
Replace NODE_NAME with the name of your node that you want to allocate GPUs from.
Configure how many GPUs to allocate to VMs. Initially, all GPUs are allocated to pods.

The total number of GPUs allocated to VMs and pods must equal the number of GPUs in the node. For example, you might have four GPUs in your node. If you allocate two GPUs to VMs, then two GPUs remain allocated to pods. The GPUAllocation manifest is rejected if you try to allocate two GPUs to VMs and one GPU to pods, as one GPU is left unallocated.

Update the number of GPUs on the node that you want to allocate for use with VMs, as shown in the following example:
```
apiVersion: gpu.cluster.gke.io/v1
kind: GPUAllocation
metadata:
  name: gpu-w2
  namespace: vm-system
spec:
  node: gpu-w2
  pod: 0
  vm: 4
```
In this example, all four GPUs installed in the node are allocated to VMs. No GPUs are allocated to pods.
Save and close the GPUAllocation custom resource in your editor.

Verify that the GPUs report their ALLOCATED status as true:

kubectl get gpuallocations --namespace vm-system

The following example output shows that the GPUs on the nodes are available for use:

NAME     ALLOCATED   DEVICEMODEL
gpu-w1   true        Tesla A100 SXM4 40GB
gpu-w2   true        Tesla A100 SXM4 40GB

Create a VM with GPU support

You can now create a VM that uses the GPU from the node. In the VM custom resource, you specify the name and quantity of GPUs to allocate from the node.

Get the name of the GPU card from the host:

kubectl describe node NODE_NAME

Replace NODE_NAME with the name of the host that you want to get the GPU name from.

The following example output shows that the allocatable GPU name on this node is NVIDIA_A100_SXM4_40GB:

Name:               bm-node1
Roles:              worker
[...]
Allocatable:
  cpu:                                         47810m
  [...]
  memory:                                      336929400Ki
  nvidia.com/gpu-vm-NVIDIA_A100_SXM4_40GB:     1
[...]

Create a VirtualMachine manifest, such as my-gpu-vm.yaml, in the editor of your choice:
```
nano my-gpu-vm.yaml
```
Copy and paste the following YAML manifest:
```
apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachine
metadata:
  name: VM_NAME
spec:
  interfaces:
    - name: eth0
      networkName: pod-network
      default: true
  disks:
    - virtualMachineDiskName: VM_NAME-boot-dv
      boot: true
      gpu:
        model: nvidia.com/gpu-vm-GPU_NAME
        quantity: 1
```
In this YAML file, define the following settings:
- VM_NAME: the name for your VM.
- GPU_NAME: the GPU name from the node to allocate to the VM.
  - This GPU name is shown in the output of the kubectl describe nodecommand from a previous step, such as NVIDIA_A100_SXM4_40GB.
The VM connects eth0 to the default pod-network network.

The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create and manage virtual disks.
Save and close the VM manifest in your editor.
Create the VM using kubectl:
```
kubectl apply -f my-gpu-vm.yaml
```
When your VM is running, connect to the VM and verify that the GPU hardware is available.

What's next

Edit a VM in Google Distributed Cloud.