Troubleshoot namespace stuck in the Terminating state

Autopilot Standard

This page shows you how to resolve issues with a namespace stuck in the Terminating state.

Namespaces use Kubernetes finalizers to prevent deletion if one or more resources within a namespace still exist. When you delete a namespace using the kubectl delete command, the namespace enters the Terminating state. The namespace stays in the Terminating state until Kubernetes deletes its dependent resources and clears all finalizers. The namespace lifecycle controller first lists all resources in the namespace that GKE needs to delete.

If GKE can't delete a dependent resource, or if the namespace lifecycle controller can't verify that the namespace is empty, then the namespace remains in the Terminating state until you resolve the issue.

To resolve a namespace stuck in the Terminating state, identify and remove any unhealthy components blocking the deletion. Try these solutions in order:

Find and remove unavailable API services.
Find and remove remaining resources.
Force delete the namespace.

Find and remove unavailable API services

List unavailable API services:
```
kubectl get apiservice | grep False
```
Troubleshoot any unresponsive services:
```
kubectl describe apiservice API_SERVICE
```
Replace API_SERVICE with the name of the unresponsive service.
Check if the status of the namespace is still Terminating:
```
kubectl get ns | grep Terminating
```

Find and remove remaining resources

List all the resources remaining in the terminating namespace:
```
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n NAMESPACE
```
Replace NAMESPACE with the name of the namespace you want to delete.
Remove any resources displayed in the output.
Check if the status of the namespace is still Terminating:
```
kubectl get ns | grep Terminating
```

Force delete the namespace

Remove the finalizers blocking namespace deletion to force the namespace to terminate:

Save the namespace manifest as a YAML file:

kubectl get ns NAMESPACE -o yaml > ns-terminating.yml

Open the manifest in a text editor and remove all values in the spec.finalizers field:
```
vi ns-terminating.yml
```

Verify that the finalizers field is empty:

cat ns-terminating.yml

The output should look similar to the following:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
  name: NAMESPACE
spec:
  finalizers:
status:
  phase: Terminating

Start an HTTP proxy to access the Kubernetes API:
```
kubectl proxy
```

Replace the namespace manifest using curl:

curl -H "Content-Type: application/yaml" -X PUT --data-binary @ns-terminating.yml http://127.0.0.1:8001/api/v1/namespaces/NAMESPACE/finalize

Check if the status of the namespace is still Terminating:
```
kubectl get ns | grep Terminating
```

What's next

If you need additional assistance, reach out to Cloud Customer Care.