Manage virtual machines on Distributed Cloud connected servers

This page describes how to manage virtual machines on Google Distributed Cloud connected servers running VM Runtime on Google Distributed Cloud. You must be familiar with VM Runtime on GDC before completing the steps on this page. For a list of supported guest operating systems, see Verified guest operating systems for VM Runtime on GDC.

To learn how virtual machines serve as an essential component of the Distributed Cloud connected platform, see Extending GKE Enterprise to manage on-premises edge VMs.

Distributed Cloud connected clusters support virtual machine webhooks. This allows Distributed Cloud connected to validate user requests made to the local Kubernetes API server. Rejected requests generate detailed information on the reason for rejection.

Configure Symcloud Storage

Google Distributed Cloud connected servers use Rakuten Symcloud Storage, as their storage solution. Symcloud Storage is a third-party solution that acts as a local storage abstraction layer on each Distributed Cloud connected node and makes its local storage available to workloads running on other Distributed Cloud connected nodes.

Symcloud Storage is deployed from Google Cloud Marketplace and is subject to the terms stated therein. Google provides limited support for using Symcloud Storage with Distributed Cloud connected and might engage the third-party provider for assistance. Software updates for Symcloud Storage are included in the Distributed Cloud connected software updates.

Configure your Google Distributed Cloud connected cluster as follows to enable Symcloud Storage for virtual machines:

  1. Create the robinio namespace with the following command:

    kubectl create ns robinio

  2. Obtain the Symcloud Storage license file and apply it to the cluster with the following command:

    kubectl apply LICENSE_FILE

  3. Verify that Symcloud Storage is up and running with the following command:

    kubectl apply LICENSE_FILE

    The command returns output similar to the following:

     Name:         robin
 Namespace:
 Labels:       app.kubernetes.io/instance=robin
               app.kubernetes.io/managed-by=robin.io
               app.kubernetes.io/name=robin
 Annotations:  <none>
 API Version:  manage.robin.io/v1
 Kind:         RobinCluster
 Metadata:
   …
 Spec:
   …
 Status:
   …
   Phase:             Ready
   …

  4. Create the robin-block-immediate storage class by applying the following configuration to the cluster:

    apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: robin-block-immediate
parameters:
  faultdomain: host
  replication: "3"
  blocksize:   "512"
provisioner: robin
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

  5. Create the robin-snapshotclass volume snapshot class by applying the following configuration to the cluster:

    apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: robin-snapshotclass
  labels:
    app.kubernetes.io/instance: robin
    app.kubernetes.io/managed-by: robin.io
    app.kubernetes.io/name: robin
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: robin
deletionPolicy: Delete

Enable VM Runtime on GDC support on Distributed Cloud connected

By default, VM Runtime on GDC virtual machine support is enabled on Distributed Cloud connected. If you need to manually enable it, complete the steps in this section. The instructions in this section assume that you have a fully functioning Distributed Cloud connected cluster. To disable VM Runtime on GDC virtual machine support on your cluster, undo the changes described in this section.

To enable the VM Runtime on GDC virtual machine subsystem, complete the following steps:

  1. Modify the VMRuntime custom resource with the following contents and apply it to your cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VMRuntime
metadata:
  annotations:
    baremetal.cluster.gke.io/vmrumtime-force-disable: "false"
    vm.cluster.gke.io/enable-vm-backup: "true"
spec:
  enabled: true
  storage:
    defaultStorageClass: robin-block-immediate
haPolicy:
  defaultRecoveryStrategy: Reschedule
  nodeHeartbeatInterval: 15s
  nodeMonitorGracePeriod: 55s

    This process typically takes several minutes to complete.

  2. Use the following command to verify that the VMRuntime custom resource has been applied to your cluster:

    kubectl get vmruntime

    The command returns output similar to the following example:

    NAME        AGE   ENABLED   READY   PREFLIGHTCHECK
vmruntime   5m   true      true    true

  3. Modify the storageprofile for the robin-block-immediate storage class with the following contents and apply it to your cluster:

    apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: robin-block-immediate
spec:
  claimPropertySets:
    accessModes:
      ReadWriteMany
  volumeMode: Block

Install the virtctl management tool

You need the virtctl client tool to manage virtual machines on your Distributed Cloud connected cluster. To install the tool, complete the following steps:

  1. Install the virtctl client tool as a kubectl plugin:

    export VERSION=v0.59.0-anthos1.28-gke.8
gcloud storage cp gs://anthos-baremetal-release/virtctl/${VERSION}/linux-amd64/virtctl/usr/local/bin/virtctl
cd /usr/local/bin
sudo ln -s virtctl kubectl-virt
sudo chmod a+x virtctl
cd -

  2. Verify that the virt plugin is installed:

    kubectl plugin list

    If the plugin has been successfully installed, the command's output lists kubectl-virt as one of the plugins.

Create a Cloud Storage bucket for virtual machine images

Complete the steps in this section to create a Cloud Storage bucket for your virtual machine images. If you already have an established image repository, skip this section.

  1. Complete the steps in Create buckets to create a bucket.

  2. Configure a service account and a key to for accessing the bucket with the following commands:

    export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create image-access
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:image-access@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer" \
gcloud iam service-accounts keys create ./image-access-gcr.json \
  --iam-account="image-access@${PROJECT_ID}.iam.gserviceaccount.com"

  3. Create a secret in the cluster for accessing the bucket. If your bucket is public, skip this step. This secret must exist in the same namespace as your virtual machine disks. You must create a secret in each affected namespace.

    kubectl create secret generic gcs-image-sa --from-file=creds-gcp.json=./image-access-gcr.json -n NAMESPACE

    Replace CLUSTER_ID with the name of the target namespace.

  4. Store your images in the bucket.

Create a virtual machine disk from a virtual machine image

Complete the steps in this section to create a virtual machine disk from a virtual machine image.

Create a disk from an image stored in Cloud Storage

Create a virtual machine disk from a virtual machine stored in your Cloud Storage bucket by applying the following configuration to your cluster:

 apiVersion: vm.cluster.gke.io/v1
 kind: VirtualMachineDisk
 metadata:
   name: DISK_NAME
   namespace: NAMESPACE
 spec:
   source:
     gcs:
       url: gs://{PROJECT_ID}-vm-images/IMAGE_FILE
       secretRef: gcs-image-sa
   size: DISK_SIZE
   storageClassName: robin-block-immediate

Replace the following:

  • DISK_NAME: the name of this virtual machine disk.
  • NAMESPACE: the target namespace.
  • IMAGE_FILE: the name of virtual machine image file.
  • DISK_SIZE: the desired disk size. This must be larger than the virtual-size value of the virtual machine image file. You can find this value with the command qemu-img info DISK_SIZE.

If you don't specify a storageClassName value, the default value specified in the VMRuntime resource is used.

Create a disk from an existing disk image

Create a virtual machine disk from an existing virtual machine disk or image file on your cluster as follows.

  1. Create the target disk by applying the following configuration to your cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineDisk
metadata:
  name: IMAGE_DISK_NAME
  namespace: NAMESPACE
spec:
  source:
    virtualMachineDisk:
      name: EXISTING_DISK_NAME
  size: DISK_SIZE
  storageClassName: robin-block-immediate

    Replace the following:

    • IMAGE_DISK_NAME: the name of this virtual machine disk.
    • NAMESPACE: the target namespace.
    • EXISTING_DISK_NAME: the name of the existing virtual machine disk.
    • DISK_SIZE: the desired disk size. This must be equal to or larger than the size of the existing disk or image file.

    You can obtain the size of your existing disk or image file using the qemu-img tool as follows:

    qemu-img info EXISTING_DISK_NAME

    If you don't specify a storageClassName value, the default value specified in the VMRuntime resource is used.

  2. Create a new disk from your existing image file by applying the following configuration to your cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineDisk
metadata:
  name: DISK_NAME
  namespace: NAMESPACE
spec:
  source:
    virtualMachineDisk:
      name: IMAGE_DISK_NAME
  size: DISK_SIZE
  storageClassName: robin-block-immediate

    Replace the following:

    • IMAGE_DISK_NAME: the name of the virtual machine disk you created in the previous step.
    • NAMESPACE: the target namespace.
    • DISK_SIZE: the desired disk size. Must be equal to the size of the virtual machine disk you created in the previous step.

  3. Resize the target disk to the desired size using the following command:

    kubectl edit gdisk DISK_NAME -n NAMESPACE

    Replace the following:

    • DISK_NAME: the name of the virtual machine disk you created in the previous step.
    • NAMESPACE: the target namespace.

  4. Modify the spec.size value in the configuration of the disk and apply it to your cluster.

Create an empty disk

Create an empty virtual machine disk by applying the following configuration to your cluster:

 apiVersion: vm.cluster.gke.io/v1
 kind: VirtualMachineDisk
 metadata:
   name: DISK_NAME
   namespace: NAMESPACE
 spec:
   size: DISK_SIZE
   storageClassName: robin-block-immediate

Replace the following:

  • DISK_NAME: the name of this virtual machine disk.
  • NAMESPACE: the target namespace.
  • DISK_SIZE: the desired disk size in gibibytes. This must be larger than the virtual-size value of the virtual machine image file. You can find this value with the command qemu-img info DISK_SIZE.

If you don't specify a storageClassName value, the default value specified in the VMRuntime resource is used.

Configure virtual networking

Follow the steps in Networking to configure the virtual networking for your virtual machines.

Create a virtual machine

Complete the steps in this section to create a virtual machine on your Distributed Cloud connected server deployment. The instructions in this section are examples meant to illustrate configurations for different scenarios. For detailed information on configuring virtual machines, see Create a VM with specific CPU and memory resources using VM Runtime on GDC.

Create a virtual machine using Google Cloud console

To create a virtual machine using Google Cloud console, do the following:

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. Select the target Google Cloud project.

  3. (Optional) If you have not already done so, log into the target cluster:

    1. In the left-hand navigation pane, click Clusters.

    2. In the Anthos Managed Clusters list, click the target cluster.

    3. In the information pane on the right-hand side, click LOG IN.

    4. In the pop-up dialog that appears, select your preferred authentication method, input your credentials, and click LOG IN.

  4. Go to the Virtual Machines page.

    Go to Virtual Machines

  5. Click CREATE.

  6. In the Basics section, do the following:

    1. In the Name field, enter a meaningful name for the virtual machine.

    2. In the Select cluster field, select the target cluster for the virtual machine.

    3. In the Namespace field, select the target namespace.

    4. In the OS Type field, select the target operating system.

    5. (Optional) If you want to add one or more labels to this virtual machine's configuration, click ADD LABEL.

  7. In the Machine Configuration section, do one of the following:

    • If you want to specify the number of vCPUs and the amount of memory for this virtual machine, select Custom Configuration and enter the target values, then click NEXT.

    • If you want to use a predetermined number of vCPUs and amount of memory for this virtual machine, select Standard Configuration and select a machine configuration from the Machine Type drop-down list, then click NEXT.

  8. In the Storage section, do one of the following:

    • If you want to create a new virtual disk for this virtual machine, select Add a new disk and enter a meaningful name in the Name field, a size in gigabytes in the GiB field, and a URL in the Image field.

    • If you want to use an existing virtual disk for this virtual machine, select Select an existing disk and select the target disk image from the Select a disk drop-down list.

    • Specify whether you want the disk to be read-only and whether it will be automatically deleted when this virtual machine is deleted using the Read-only and Auto delete checkboxes.

    • Specify a virtual disk driver in the Driver drop-down list.

    • To add another disk to this virtual machine, click ADD A DISK in the Additional disks section.

    • Click NEXT.

  9. In the Network section, do the following:

    1. In the Default network interface subsection, specify the name of the primary network interface for this virtual machine in the Interface name field.

    2. Select the corresponding network type from the Network type drop-down list.

    3. Specify whether to allow external access to this network interface using the Allow external access checkbox. If you enable this option, you must enter a comma-separated list of ports to expose externally in the Exposed ports field.

    4. If you want to add one or more secondary network interfaces to this virtual machine, click ADD A NETWORK INTERFACE.

    5. Click NEXT.

  10. In the Advanced options section, use the Auto-restart on update checkbox to specify whether to restart this virtual machine after the
    Distributed Cloud connected software on the target cluster has been updated.

  11. In the Firmware section, do the following:

    1. Select the target firmware type in the Bootloader type field. If you select UEFI firmware, you have the option to enable secure boot using the Secure boot checkbox.

    2. Specify a serial number for this virtual machine in the Serial field.

    3. Specify a Universally Unique Identifier (UUID) for this virtual machine in the UUID field.

  12. In the Cloud-init section, do the following:

    • Specify a network data secret value in the Network data secret field.

    • Specify a user data secret value in the User data secret field.

  13. Click CREATE VM to create the virtual machine.

Create a virtual machine from a YAML file

To create a virtual machine from a YAML configuration file, do the following:

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. Select the target Google Cloud project.

  3. (Optional) If you have not already done so, log into the target cluster:

    1. In the left-hand navigation pane, click Clusters.

    2. In the Anthos Managed Clusters list, click the target cluster.

    3. In the information pane on the right-hand side, click LOG IN.

    4. In the pop-up dialog that appears, select your preferred authentication method, input your credentials, and click LOG IN.

  4. Go to the Virtual Machines page.

    Go to Virtual Machines

  5. Click CREATE WITH YAML.

  6. In the Select cluster field, select the target cluster for the virtual machine.

  7. Paste the virtual machine configuration in YAML format into the YAML field.

  8. Click CREATE.

Create a virtual machine from a bootable disk image

To create a virtual machine from a bootable disk image, apply the following configuration to your cluster:

kind: VirtualMachine
metadata:
  name: my-virtual-machine
  namespace: my-vm-namespace
spec:
  osType: Linux/Windows
  guestEnvironment: {} // comment out this line to enable guest environment for access management
  autoRestartOnConfigurationChange: true 
  compute:
    cpu:
      vcpus: 6
    memory:
      capacity: 8Gi
  interfaces:
      - name: eth0
        networkName: network-410
        ipAddresses:
        - 10.223.237.10/25
  disks:
    - virtualMachineDiskName: my-boot-disk
      boot: true
    - virtualMachineDiskName: my-data-disk

Replace the following:

  • DISK_NAME: the name of this virtual machine disk.
  • NAMESPACE: the target namespace.

Create a virtual machine from an ISO optical disc image

To create a virtual machine from an ISO optical disc image, complete the steps in Create a Windows VM from ISO image in Google Distributed Cloud.

Create a virtual machine with GPU support

Complete the steps described on this page to create a virtual machine configured to meet your business requirements, then complete the steps in Configure a virtual machine to use GPU resources.

Access a virtual machine

Complete the steps in this section to access a virtual machine running on your Distributed Cloud connected server deployment.

Obtain access credentials

Complete the steps in this section to obtain the credentials necessary to access your virtual machine using the Linux guest environment feature.

  1. Enable the Linux guest environment for the target virtual machine by applying the following configuration to your cluster:

    kind: VirtualMachine
metadata:
 name: my-virtual-machine
 namespace: my-vm-namespace
spec:
 osType: Linux
 guestEnvironment: {}
 autoRestartOnConfigurationChange: true

  2. Generate an id_rsa.pub file containing an SSH key pair with the following command:

    ssh-keygen -t rsa

  3. Create a VirtualMachineAccessRequest resource by applying the following configuration to your cluster:

    apiVersion: vm.cluster.gke.io/v1alpha1
kind: VirtualMachineAccessRequest
metadata:
 name: RESOURCE_NAME
 namespace: NAMESPACE
spec:
  vm: VM_NAME
  user: USER_NAME
  ssh:
    key: RSA_KEY
    ttl: 2h

Replace the following:

  • RESOURCE_NAME: a descriptive name for this virtual machine access request resource.
  • NAMESPACE: the target namespace.
  • VM_NAME: the name of the target virtual machine.
  • USER_NAME: the name of the user to whom access is being granted.
  • RSA_KEY: the contents of the id_rsa.pub file you generated in the previous step.

  1. Check the status of the access request with the following command:

    kubectl get vmar

    When the command returns a Configured status, proceed to the next step.

  2. Access the virtual machine with SSH or Remote Desktop:

    • If the virtual machine is connected to your local network, you can access it directly.
    • If the virtual machine is connected to the Pod network, you must create a load balancer service to access the required ports.

Start, restart, or stop a virtual machine

Use the following commands to start, restart, or stop a virtual machine:

  • Start a virtual machine: kubectl virt start vmVM_NAME-nNAMESPACE
  • Restart a virtual machine: kubectl virt restart vmVM_NAME-nNAMESPACE
  • Stop a virtual machine: kubectl virt stop vmVM_NAME-nNAMESPACE

Replace the following:

  • VM_NAME: the name of the target virtual machine.
  • NAMESPACE: the target namespace.

Start or stop a virtual machine using Google Cloud console

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. Select the target Google Cloud project.

  3. (Optional) If you have not already done so, log into the target cluster:

    1. In the left-hand navigation pane, click Clusters.

    2. In the Anthos Managed Clusters list, click the target cluster.

    3. In the information pane on the right-hand side, click LOG IN.

    4. In the dialog that appears, select your preferred authentication method, input your credentials, and click LOG IN.

  4. Go to the Virtual Machines page.

    Go to Virtual Machines

  5. In the list of virtual machines, select the checkbox(es) for the target virtual machine(s).

  6. At the top of the page, click START or STOP as applicable.

View the status of a virtual machine using Google Cloud console

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. Select the target Google Cloud project.

  3. (Optional) If you have not already done so, log into the target cluster:

    1. In the left-hand navigation pane, click Clusters.

    2. In the Anthos Managed Clusters list, click the target cluster.

    3. In the information pane on the right-hand side, click LOG IN.

    4. In the pop-up dialog that appears, select your preferred authentication method, input your credentials, and click LOG IN.

  4. Go to the Virtual Machines page.

    Go to Virtual Machines

  5. Click the target virtual machine.

  6. In the page that appears, click the Details, Events, and YAML tab to view the corresponding information about this virtual machine.

Modify a virtual machine

To modify a virtual machine, you must delete it and re-create it with the updated configuration.

Delete a virtual machine using Google Cloud console

  1. In the Google Cloud console, go to the Clusters page.

    Go to Clusters

  2. Select the target Google Cloud project.

  3. (Optional) If you have not already done so, log into the target cluster:

    1. In the left-hand navigation pane, click Clusters.

    2. In the Anthos Managed Clusters list, click the target cluster.

    3. In the information pane on the right-hand side, click LOG IN.

    4. In the dialog that appears, select your preferred authentication method, input your credentials, and click LOG IN.

  4. go to the Virtual Machines page.

    Go to Virtual Machines

  5. In the list of virtual machines, select the checkbox for the target virtual machine.

  6. At the top of the page, click DELETE.

  7. In the confirmation dialog that appears, enter the virtual machine's name and click DELETE.

Back up a virtual machine

This section describes how to configure your environment for backing up your virtual machine workloads and how to manage your backups. To enable this feature, contact your Google representative.

Prerequisites

  1. If you have not already done so, create the robin-block-immediate custom resource of type StorageClass with the following contents and apply it to your cluster:

    apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: robin-block-immediate
 annotations:
   storageclass.kubernetes.io/is-default-class: "true"
parameters:
 faultdomain: host
 replication: "3"
 blocksize:   "512"
provisioner: robin
reclaimPolicy: Delete
volumeBindingMode: Immediate

  2. If you have not already done so, create a robin-snapshotclass custom resource of type VolumeSnapshotClass with the following contents and apply it to your cluster:

    apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
 name: robin-snapshotclass
 labels:
   app.kubernetes.io/instance: robin
   app.kubernetes.io/managed-by: robin.io
   app.kubernetes.io/name: robin
 annotations:
   snapshot.storage.kubernetes.io/is-default-class: "true"
driver: robin
deletionPolicy: Delete

Create an online backup repository

An online backup repository is an S3-compatible online storage location for your virtual machine backups. It also stores records of backups, backup plans, restore plans, and serves as a destination for restoring virtual machine backups.

Complete the following steps to create an online backup repository for your virtual machines:

  1. Create a Cloud Storage bucket as described in Create buckets.

  2. Create a service account and key to access the bucket using the following commands:

    export PROJECT_ID=$(gcloud config get-value project)
export SVC_ACCOUNT=backup-access
gcloud iam service-accounts create $SVC_ACCOUNT
gcloud projects add-iam-policy-binding $PROJECT_ID \ 
   --member="serviceAccount:${SVC_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \
   --role="roles/backupdr.cloudStorageOperator"
gcloud storage hmac create  ${SVC_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com > hmac_temp_key_file
awk 'NR==1{print "ACCESS_ID=" $NF} NR==2{print "SECRET=" $NF}' < hmac_temp_key_file > hmac_key_file

  3. Configure the secret for accessing the backup bucket using the following commands:

    source ./hmac_key_file
kubectl create secret generic SECRET_NAME \
   --from-literal=access-key=$SECRET \
   --from-literal=access-key-id=$ACCESS_ID -n NAMESPACE

    Replace the following:

    • SECRET_NAME: a descriptive name for this secret.
    • NAMESPACE: the target namespace.

  4. Create the BackupRepository resource applying the following configuration to your cluster:

    apiVersion: backup.gdc.goog/v1
kind: BackupRepository
metadata:
  name: "REPOSITORY_NAME"
spec:
  secretReference:
    namespace: NAMESPACE
    name: gcs-hmac-secret
  endpoint: "https://storage.googleapis.com"
  type: "S3"
  s3Options:
    bucket: $BUCKET_NAME 
    region: "REGION"
    forcePathStyle: true
  importPolicy: "ReadWrite"
# Force attachment for convenience.
force: true

    Replace the following:

    • REPOSITORY_NAME: a descriptive name for the repository.
    • BUCKET_NAME: the name of the backup bucket.
    • NAMESPACE: the target namespace.
    • REGION: the Google Cloud region in which the target Distributed Cloud cluster has been created.

  5. Verify that the cluster can access the backup repository using the following command:

    kubectl get BackupRepository

Create a local backup repository

A local backup repository is an S3-compatible storage location for your virtual machine backups that resides locally on your Distributed Cloud connected cluster. A local backup repository is functionally identical to an online backup repository.

  1. Generate an AES256 encryption key to use for application-level config encryption:

    openssl rand -base64 32

    The command outputs a base64-encoded random 256-bit key. For example:

    aBcD_eFgH1iJkLmN0pQrStUvWxFyZgAhIjKlMnOpQ=

  2. Double-encode the key to use as a payload in the YAML config for the BackupRepository resource:

    echo -n "AES_KEY" | base64

    Replace AES_KEY with the base64-encoded AES256 key you generated in the previous step. Store this key in a local file.

  3. Configure the secret for accessing the backup repository using the following commands:

    kubectl create secret generic SECRET_NAME \
   --from-literal=access-key=ENCODED_AES_KEY \
   --namespace NAMESPACE

    Replace the following:

    • ENCODED_AES_KEY: the double-encoded AES256 key you generated in the previous step.
    • SECRET_NAME: a descriptive name for this secret.
    • NAMESPACE: the target namespace.

  4. Configure the BackupRepository by applying the following configuration to your cluster:

    apiVersion: backup.gdc.goog/v1
kind: BackupRepository
metadata:
  name: REPOSITORY_NAME
spec:
  force: true
  importPolicy: ReadWrite
  localOptions:
    encryptionKey:
      name: SECRET_NAME
      namespace: NAMESPACE
  type: Local

  5. Verify that the cluster can access the backup repository using the following command:

    kubectl get BackupRepository

    Replace the following:

    • REPOSITORY_NAME: a descriptive name for the repository.
    • SECRET_NAME: the name of the Kubernetes secret you created in the previous step.
    • NAMESPACE: the namespace in which you created the Kubernetes secret.

Create a backup plan

A backup plan defines the automated scheduling for running your virtual machine backup. Create a VirtualMachineBackupPlan resource with the following contents and apply to your cluster:

apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineBackupPlan
metadata:
  name: BACKUP_PLAN_NAME
  namespace: NAMESPACE
spec:
  backupConfig:
    backupRepository: REPOSITORY_NAME
    backupScope:
      selectedVirtualMachines:
      - resourceName: VM_NAME
    volumeStrategy: LocalSnapshotOnly
  backupSchedule:
    cronSchedule: "CRON_SCHEDULE"
    paused: PAUSED
  retentionPolicy:
    backupDeleteLockDays: PLAN_LOCK_LENGTH
    backupRetainDays: 4
    locked: RETENTION_POLICY_LOCKED
  numBackupsToRetain: BACKUPS_RETAINED

Replace the following:

  • BACKUP_PLAN_NAME: a descriptive name for the backup plan.
  • NAMESPACE: the name of the target namespace.
  • REPOSITORY_NAME: the target backup repository.
  • CRON_SCHEDULE: cron-standard schedule for running the backups. The minimum permitted interval between backups is 10 minutes.
  • PAUSED: specifies whether this backup plan is paused. Valid values are true and false.
  • VM_NAME: specifies a virtual machine workload to back up through this backup plan. You can specify multiple virtual machine resources per backup plan.
  • * BACKUP_LOCK_LENGTH: specifies the number of days after backup creation during which the backup cannot be deleted.
  • * BACKUP_RETENTION_LENGTH: specifies the number of days to retain this backup. When the retention period elapses, the backup is deleted. If omitted, defaults to 4.
  • * RETENTION_POLICY_LOCKED: specifies whether this backup plan's retention policy is locked. Valid values are true and false.
  • * BACKUPS_RETAINED/var>: specifies the number of backups to retain in this backup plan. When this threshold is reached, backups are deleted in the order from oldest to newest.

List existing backup plans

To list existing backup plans, use the following command:

kubectl get VirtualMachineBackupPlans -A

The command returns output similar to the following:

NAMESPACE      NAME                                LASTBACKUPTIME   LASTBACKUPSTATE   NEXTBACKUPTIME   PAUSED
vm-workloads   bkp-template-vm-windows-vm-local
vm-workloads   sched-snapshotonly-bkp-plan-10min                                                       false

Create a manual backup of a virtual machine

Complete the steps in this section to create a manual backup of a virtual machine.

  1. Create a VirtualMachineBackupPlanTemplate resource in the namespace in which the target virtual machine resides by applying the following configuration to the cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineBackupPlanTemplate
metadata:
  name: TEMPLATE_NAME
  namespace: NAMESPACE
spec:
  backupRepository: REPOSITORY_NAME

    Replace the following:

    • TEMPLATE_NAME: a descriptive name for this backup template.
    • REPOSITORY_NAME: the name of the target backup repository.
    • NAMESPACE: the target namespace.

  2. Trigger the backup by creating a VirtualMachineBackupRequest resource with the following configuration and applying it to the cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineBackupRequest
metadata:
  name: BACKUP_REQUEST_NAME
  namespace: NAMESPACE
spec:
  vmBackupPlanTemplate: TEMPLATE_NAME
  virtualMachine: VM_NAME
  vmBackupName: BACKUP_NAME

    Replace the following:

    • BACKUP_REQUEST_NAME: a descriptive name for this backup request.
    • TEMPLATE_NAME: the name of the backup template you created in the previous step.
    • NAMESPACE: the name of the target namespace.
    • VM_NAME: the name of the target virtual machine.
    • BACKUP_NAME: a descriptive name for this backup.

  3. Verify the integrity of the backup with the following command:

    kubectl get vmbackup BACKUP_NAME -n NAMESPACE

    Replace the following:

    • BACKUP_NAME: the name of the target backup.
    • NAMESPACE: the target namespace.

    The command returns output similar to the following:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineBackup
metadata:
  creationTimestamp: "2024-04-09T17:57:44Z"
  finalizers:
  - vm.cluster.gke.io/virtual-machine-backup-finalizer
  generation: 1
  name: vmt13-backup-0409-2
  namespace: default
  ownerReferences:
  - apiVersion: backup.gdc.goog/v1
    kind: Backup
    name: vmt13-backup-0409-2
    uid: 0ee0b92c-1e27-48cc-8f8f-5606ea925e88
  resourceVersion: "36192759"
  uid: e471f8c7-637c-485f-acda-108017a5638f
spec:
  backupConfig:
    backupRepository: default
    backupScope:
      selectedVirtualMachines:
      - resourceName: vm-t13
    volumeStrategy: Portable
  vmBackupPlan: MyVmPlan-vm-vm-t13-portable
status:
  backedUpVirtualMachineDisks:
  - vm-t13-boot-disk
  - vm-t13-data-disk
  backedUpVirtualMachines:
  - vm-t13
  backup: vmt13-backup-0409-2
  backupStatus:
    clusterMetadata:
      k8sVersion: "1.28"
    completeTime: "2024-04-09T18:07:36Z"
    createTime: "2024-04-09T17:57:44Z"
    jobCreated: true
    resourceCount: 849
    sizeBytes: 1948672
    state: Succeeded

List virtual machine backups

To view existing virtual machine backups, use the following command:

kubectl get VirtualMachineBackups -A

The command returns output similar to the following:

NAMESPACE      NAME        STATE       CREATETIME
vm-workloads   vm-backup   Succeeded   2025-04-08T04:37:32Z

Restore a virtual machine from a backup

Complete the steps in this section to restore a virtual machine from a backup.

  1. Trigger the restore process by creating a VirtualMachineRestoreRequest resource with the following configuration and applying it to your cluster:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineRestoreRequest
metadata:
  name: restore-req
  namespace: NAMESPACE
spec:
  vmBackup: BACKUP_NAME
  restoreName: RESTORE_NAME
  restoredResourceName: RESTORED_VM_NAME
  restoredResourceDescription: RESTORE_DESCRIPTION

    Replace the following:

    • BACKUP_NAME: the name of the target backup.
    • RESTORE_NAME: a descriptive name for this restore operation.
    • NAMESPACE: the target namespace.
    • RESTORED_VM_NAME: the name given to the virtual machine during restoration. This name must not conflict with any virtual machines already present on the cluster.
    • RESTORE_DESCRIPTION: a descrpition for this restore operation.

  2. Use the following command to check the progress of the restore operation:

    kubectl get virtualmachinerestores.vm.cluster.gke.io RESTORE_NAME -n NAMESPACE

    Replace the following:

    • RESTORE_NAME: the name of the target restore operation.
    • NAMESPACE: the target namespace.

    The command returns output similar to the following:

    apiVersion: vm.cluster.gke.io/v1
kind: VirtualMachineRestore
metadata:
  creationTimestamp: "2024-04-09T18:09:51Z"
  finalizers:
  - vm.cluster.gke.io/virtual-machine-restore-finalizer
  generation: 1
  name: vmt13-restore-0409-2-1
  namespace: default
  ownerReferences:
  - apiVersion: backup.gdc.goog/v1
    kind: Restore
    name: vmt13-restore-0409-2-1
    uid: 4ce1ca83-eba0-4cc3-bad3-af6cf9185d7d
  resourceVersion: "36194596"
  uid: aba50b59-e18d-4687-ad11-47baa45478b4
spec:
  targetVirtualMachineDisks:
  - vm-t13-boot-disk
  - vm-t13-data-disk
  targetVirtualMachines:
  - vm-t13
  vmBackup: vmt13-backup-0409-2
status:
  restore: vmt13-restore-0409-2-1
  restoreStatus:
    completeTime: "2024-04-09T18:10:00Z"
    jobCreated: true
    resourcesRestoredCount: 5
    restoredVolumesCount: 2
    startTime: "2024-04-09T18:09:51Z"
    state: Succeeded
    stateReason: restore is successful

View restore operations

To view restore operations initiated to date, use the following command:

kubectl get VirtualMachineRestore.vm.cluster.gke.io -A

The command returns output similar to the following:

NAMESPACE      NAME        STARTTIME              RESTORE     STATE
vm-workloads   restore-1   2025-04-08T04:41:04Z   restore-1   Succeeded

Delete a virtual machine backup

To delete a virtual machine backup, create a VirtualMachineDeleteBackupRequest resource with the following configuration and apply it to your cluster:

    apiVersion: vm.cluster.gke.io/v1
    kind: VirtualMachineDeleteBackupRequest
    metadata:
      name: vmdbr
      namespace: BACKUP_NAME
    spec:
      vmBackup: NAMESPACE

Replace the following:

  • NAMESPACE: the name of the target namespace.
  • BACKUP_NAME: the name of the target backup.

View AppArmor sandboxing audit logs

Distributed Cloud connected automatically sandboxes virtual machine workloads with AppArmor policies in audit-mode. A policy violation emits an audit log entry. For example:

{
  "jsonPayload": {
    "_SOURCE_REALTIME_TIMESTAMP": "1734596844149104",
    "SYSLOG_TIMESTAMP": "Dec 19 08:27:24 ",
    "MESSAGE": "type=AVC msg=audit(1734596844.148:27742): apparmor=\"ALLOWED\" operation=\"open\" profile=\"virt-launcher-audit\" name=\"/etc/libvirt/virtlogd.conf\" pid=182406 comm=\"virtlogd\" requested_mask=\"r\" denied_mask=\"r\" fsuid=0 ouid=0 FSUID=\"root\" OUID=\"root\"",
    "PRIORITY": "6",
    ...
    "SYSLOG_RAW": "<14>Dec 19 08:27:24 audisp-syslog: type=AVC msg=audit(1734596844.148:27742): apparmor=\"ALLOWED\" operation=\"open\" profile=\"virt-launcher-audit\" name=\"/etc/libvirt/virtlogd.conf\" pid=182406 comm=\"virtlogd\" requested_mask=\"r\" denied_mask=\"r\" fsuid=0 ouid=0 FSUID=\"root\" OUID=\"root\"\n",
    "SYSLOG_IDENTIFIER": "audisp-syslog",
    "_GID": "0",
  },
  "timestamp": "2024-12-19T08:27:24.149109Z",
  "labels": {
    "gke.googleapis.com/log_type": "system"
  },
  "receiveTimestamp": "2024-12-19T08:27:24.721842807Z"
  ...
  ...
}

What's next