Troubleshoot storage errors in Backup for GKE

This page describes storage-related errors you might experience when using Backup for GKE, things to consider when performing the action, and steps on how to troubleshoot the issue.

Error 100010105: Failed to backup PersistentVolumeClaim - Disk referenced by PersistentVolume does not exist

Error 100010105 occurs when an attempt to back up a PersistentVolumeClaim fails because it references a disk that doesn't exist, resulting in an error message stating Failed to backup PersistentVolumeClaim - Disk referenced by PersistentVolume does not exist.

In Google Kubernetes Engine, PersistentVolumeClaims request storage from PersistentVolumes. A PersistentVolume, in turn, represents a piece of storage, often an underlying Compute Engine Persistent Disk. An error can occur when a PersistentVolumeClaim is bound to a PersistentVolume and the PersistentVolume's configuration specifies a Compute Engine Persistent Disk. However, the actual disk with the name and location specified in the PersistentVolume configuration cannot be found in your Google Cloud project. Thus, Backup for GKE cannot proceed with backing up a non-existent disk, and a failure occurs.

To resolve this error, use the following instructions:

  1. Identify the problematic PersistentVolumeClaim and PersistentVolume. The names of both the problematic PersistentVolumeClaim and its associated PersistentVolume are listed in the state reason field of your failed Backup for GKE operation. We recommend documenting both the PersistentVolumeClaim name, its namespace, and the name of the PersistentVolume.

  2. Inspect the PersistentVolume. To describe the PersistentVolume, use the PersistentVolume name you identified from the state reason field in the following command:

    kubectl describe pv PERSISTENTVOLUME_NAME

    Replace PERSISTENTVOLUME_NAME with the name of your PersistentVolume.

  3. In the output, examine the source section, specifically under csi. This section describes the VolumeHandle that the PersistentVolume is trying to reference. For example:

    Source:
  Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: my-non-existent-disk
FSType: ext4
Partition: 0
ReadOnly: false
            In this example, the PD name is my-non-existent-disk.

    Source:
  Type:       CSI (a Container Storage Interface (CSI) volume)
  Driver:     pd.csi.storage.gke.io
  VolumeHandle: projects/PROJECT_ID/zones/ZONE/disks/DISK_NAME
...

    In this example, the VolumeHandle contains the full path to the disk, including its name and location. For example, projects/my-gcp-project/zones/us-central1-a/disks/my-disk-name.

  4. Use the VolumeHandle obtained from the PersistentVolume description to identify the disk name and zone.

  5. Verify that the disk exists in your Google Cloud project by using one of the following methods:

    Zonal disk

    If you're using a zonal disk, use Google Cloud CLI to run the gcloud compute disks describe command:

    gcloud compute disks describe DISK_NAME \
    --zone=ZONE_NAME \
    --project=PROJECT_ID

    Replace the following:

    • DISK_NAME: the name of the disk you obtained from the PersistentVolume description.

    • ZONE_NAME: the zone of the disk you obtained from the PersistentVolume description.

    • PROJECT_ID: your Google Cloud project ID.

    Regional disk

    If you're using a regional disk, use Google Cloud CLI to run the gcloud compute disks describe command:

    gcloud compute disks describe DISK_NAME \
    --region=REGION_NAME \
    --project=PROJECT_ID

    Replace the following:

    • DISK_NAME: the name of the disk you obtained from the PersistentVolume description.

    • REGION_NAME: the region of the disk you obtained from the PersistentVolume description.

    • PROJECT_ID: your Google Cloud project ID.

    If you get a Resource not found or The resource DISK_NAME was not found error message, the disk doesn't exist. Use one of the following methods to resolve the issue depending on the scenario that best fits your needs:

    • If the disk was accidentally deleted or misnamed and you want to keep the data or PersistentVolumeClaim, or the PersistentVolume was configured with an incorrect disk name, use one of the following methods to resolve the issue:

      • Restore the disk: if you have a backup of the disk, restore it with the exact same name and location that the PersistentVolume references.

      • Create a new disk: if restoring the disk isn't an option, create a new disk with the same name and location that's in the PersistentVolume configuration.

    • If the PersistentVolumeClaim or PersistentVolume, their data, or the application are no longer needed, we recommend removing the unneeded entity:

      • Delete the PersistentVolumeClaim: delete the PersistentVolumeClaim by using the kubectl command-line tool to run the kubectl delete pvc command:
      kubectl delete pvc PVC_NAME -n NAMESPACE

      Replace the following:

      • PVC_NAME: the name of the PersistentVolumeClaim you want to delete.

      • NAMESPACE: the namespace of the PersistentVolumeClaim you want to delete.

    • The PersistentVolume is still present after you delete the PersistentVolumeClaim: if the PersistentVolume's PersistentVolumeReclaimPolicy is set to Delete, the PersistentVolume is automatically deleted when the PersistentVolumeClaim is deleted. If the persistentVolumeReclaimPolicy is set to Retain, you need to manually delete the PersistentVolume after the PersistentVolumeClaim is deleted. To delete the PersistentVolume, use the kubectl command-line tool to run the kubectl delete pv command:

      kubectl delete pv PV_NAME

      Replace PV_NAME with the name of the PersistentVolume you want to delete.

If the operation continues to fail, contact Cloud Customer Care for further assistance.