Troubleshoot system errors in Backup for GKE

Autopilot Standard

This page describes system-related errors you might experience when you use Backup for GKE, things to consider when you back up resources, and steps for troubleshooting the issue.

Error 100020102: Strict permissive mode - Failed to backup CRD - Unsupported `v1beta1` API version

Error 100020102 occurs when an attempt to back up a CustomResourceDefinition that was originally applied as a apiextensions.k8s.io/v1beta1 version fails because it lacks the structural schema that's required in the apiextensions.k8s.io/v1 API version. This error results in the following error message: Strict permissive mode - Failed to backup CRD - Unsupported v1beta1 API Version.

This error occurs because the apiextensions.k8s.io/v1 API version was removed in Google Kubernetes Engine version 1.22. For more information about the API removal for GKE version 1.22, see API removals for GKE v1.22.

Backup operation behavior in non-permissive mode

In non-permissive mode, or in a strict backup plan, the backup operation fails if it encounters a resource that can't be backed up, such as a CustomResourceDefinition that was created with the v1beta1 API. This error occurs because the resource lacks the structural schema that's required by the v1 API. The presence of this CustomResourceDefinition is considered a critical error because it might not restore correctly to a newer cluster.

To resolve this error, use the following instructions:

Identify the problematic CustomResourceDefinition by running the kubectl get crd command:
```
kubectl get crd CRD_NAME
```
Replace CRD_NAME with the name of the CustomResourceDefinition from your error message.
In the YAML output, confirm whether the CustomResourceDefinition was properly converted from the vbeta1 API to the v1 API by locating the following conditions:
1. spec.versions: locate the spec.versions condition by looking through each version that's listed under the spec.versions field. If any of the spec.versions are missing the schema.openAIV3Schema field, the CustomResourceDefinition doesn't have a structural schema defined for that version.
2. status.conditions: locate the status.conditions condition by finding the type:NonStructuralSchema condition. If the status.conditions's status is true, it explicitly confirms that the schema is not structural.
Upgrade the CustomResourceDefinition to v1 API version by using the following steps:
1. Edit the existing CustomResourceDefinition to make it compatible with the v1 standard by adding a structural schema that defines every field and its type within the custom resource. For more information about how to add a structural schema, see Specifying a structural schema.
2. Apply the compatible v1 manifest to your cluster.
  
  Note: In some cases, you might need to delete the previously created CustomResourceDefinition that lacks a structural schema and re-apply the v1 version so that the API server correctly handles the resource convention.
If the upgrade succeeds, re-attempt the backup operation. Otherwise, use one of the following methods to resolve the issue:
- Delete the CustomResourceDefinition by running the kubectl delete crd command if the CustomResourceDefinition isn't being used in the cluster.
  
  Note: You shouldn't attempt to delete a CustomResourceDefinition if it's actively serving an existing custom resource in the cluster. You either need to attempt to upgrade the CustomResourceDefinition to v1 API version again or enable permissive mode.
```
kubectl delete crd CRD_NAME
```
  Replace CRD_NAME with the name of the CustomResourceDefinitionyou want to delete.
- Enable permissive mode on the backup plan, which allows Backup for GKE to skip the resource—including CustomResourceDefinitions in the v1beta1 API version—and continue with the rest of the backup operation. For more information about how to enable permissive mode, see Enable permissive mode on a backup plan.
  
  Warning: The problematic CustomResourceDefinition and its associated custom resources won't be backed up and won't be included when you restore. To successfully restore such a backup, you need to exclude the resources served by the CustomResourceDefinition from the restore, or create the CustomResourceDefinition in the target cluster before you start the restore.
Re-attempt the backup operation. If the operation continues to fail, contact Cloud Customer Care for further assistance.

Error 100040102: Namespace not found

Error 100040102 occurs when an attempt to perform a backup operation fails because a namespace specified in the backup scope can't be found within the cluster. The Backup for GKE agent wasn't able to locate one or more namespaces that were explicitly listed in the selectedNamespaces field of the BackupPlan configuration. Backup for GKE requires all specified namespaces to be present in the cluster at the time the backup operation is initiated. Failure to find the namespace results in the following error message:

Namespace [NAMESPACE_NAME] is not found.

To resolve this issue, use the following instructions:

Verify that the namespace has been entered correctly by checking the selectedNamespaces list in your BackupPlan configuration.
Confirm that the namespace reported in the error message exists by running the kubectl get namespace command:
```
kubectl get namespace NAMESPACE_NAME
```
Replace NAMESPACE_NAME with the name of the namespace reported in the error message.

If the namespace doesn't exist, a message appears stating that the namespace wasn't found, for example, Error from server (NotFound): namespaces "[NAMESPACE_NAME]" not found.
Correct the BackupPlan. If the namespace was misspelled, update the BackupPlan with the correct namespace name. If the namespace truly no longer exists and does not need to be backed up, remove it from the selectedNamespaces list in the BackupPlan configuration.
Re-attempt the backup operation after making the necessary corrections to the BackupPlan and initiate a new backup.

If the operation continues to fail, contact Cloud Customer Care for further assistance.

What's next

Read the Backup for GKE error codes overview page.