Troubleshoot Google Distributed Cloud webhooks issues
Stay organized with collections
Save and categorize content based on your preferences.
This page shows you how to resolve issues with problematic or unsafe webhooks in
Google Distributed Cloud.
Types of problematic webhooks
Admission webhooks, or webhooks in Kubernetes, are a type of
admission controller
that can be used in Kubernetes clusters to validate or mutate requests to the
control plane prior to a request being persisted. It is common for third-party
applications to use webhooks that operate on system-critical resources and
namespaces. Incorrectly configured webhooks can impact control plane
performance and reliability. For example, an incorrectly configured webhook
created by a third-party application could prevent Google Distributed Cloud from
creating and modifying resources in the managed kube-system namespace, which
could degrade the functionality of the cluster.
If a webhook has no available endpoints, the Service that backs the webhook
endpoint has one or more Pods which aren't running. To make the webhook
endpoints available, follow the instructions to find and troubleshoot the Pods
of the Service that is backing this webhook endpoint:
Find the serving Pods for the Service associated with the webhook. Run the
following command to describe the Service:
kubectldescribesvcSERVICE_NAME-nSERVICE_NAMESPACE
Replace the following:
SERVICE_NAME with the name of the Service.
SERVICE_NAMESPACE with the name of the namespace.
If you can't find the Service name listed in the webhook, the unavailable
endpoint might be caused by a mismatch between the name listed in the
configuration and the actual name of the Service. To fix the endpoint
availability, update the Service name in the webhook configuration to match
the correct Service object.
Inspect the serving Pods for this Service. Identify which Pods aren't
running by listing the Deployment:
kubectlgetdeployment-nSERVICE_NAMESPACE
Or, run the following command to list the Pods:
kubectlgetpods-nSERVICE_NAMESPACE-owide
For any Pods that aren't running, inspect the Pod logs to see why the Pod
isn't running.
Webhooks that are considered unsafe
If a webhook intercepts any resources in system-managed namespaces, we
recommend that you update the webhooks to avoid intercepting these resources.
Inspect the webhook configuration. Run the following kubectl command to
get the webhook configuration:
Replace CONFIGURATION_NAME with the name of the
webhook configuration.
If this command doesn't return anything, run the command again, replacing
validatingwebhookconfigurations with mutatingwebhookconfigurations.
In the webhooks section of the output, one or more webhooks are listed.
Edit the configuration, depending on the reason the webhook is considered
unsafe:
Exclude kube-system and kube-node-lease namespaces
A webhook is considered unsafe if scope is *, or if scope is
Namespaced and either of the following conditions are true:
The operator condition is NotIn and values omits kube-system and
kube-node-lease, as in the following example:
webhooks:-admissionReviewVersions:...namespaceSelector:matchExpressions:-key:kubernetes.io/metadata.nameoperator:NotInvalues:-blue-system# add 'kube-system' and 'kube-node-lease' if `NotIn`objectSelector:{}rules:-apiGroups:...scope:'*'# 'Namespaced'sideEffects:NonetimeoutSeconds:3
Ensure that scope is set to Namespaced, not *, so that the webhook
only operates in specific namespaces. Ensure that if operator is
NotIn, kube-system and kube-node-lease are included in values.
The operator condition is In and values includes kube-system and
kube-node-lease, as in the following example:
namespaceSelector:matchExpressions:-key:kubernetes.io/metadata.nameoperator:Invalues:-blue-system-kube-system# remove as operator is `In`-kube-node-lease# remove as operator is `In`
Ensure that scope is set to Namespaced, not *, so that the webhook
only operates in specific namespaces. Ensure that if operator is In,
kube-system and kube-node-lease are not included in values.
Exclude matched resources
A webhook is also considered unsafe if nodes, tokenreviews,
subjectaccessreviews, or certificatesigningrequests are listed under
resources, as in the following example:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[],[],null,["# Troubleshoot Google Distributed Cloud webhooks issues\n\nThis page shows you how to resolve issues with problematic or unsafe webhooks in\nGoogle Distributed Cloud.\n\nTypes of problematic webhooks\n-----------------------------\n\nAdmission webhooks, or *webhooks* in Kubernetes, are a type of\n[admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)\nthat can be used in Kubernetes clusters to validate or mutate requests to the\ncontrol plane prior to a request being persisted. It is common for third-party\napplications to use webhooks that operate on system-critical resources and\nnamespaces. Incorrectly configured webhooks can impact control plane\nperformance and reliability. For example, an incorrectly configured webhook\ncreated by a third-party application could prevent Google Distributed Cloud from\ncreating and modifying resources in the managed `kube-system` namespace, which\ncould degrade the functionality of the cluster.\n\nProblematic webhooks include the following types:\n\n- **Webhooks that operate, but have no endpoints available.** Follow the instructions to [check webhooks with no available endpoints](#no-available-endpoints).\n- **Webhooks that are considered unsafe as they operate on system critical\n resources and namespaces.**\n\n The following webhooks are considered unsafe:\n - Webhooks that intercept Pods and leases in the `kube-system` namespace.\n - Webhooks that intercept leases in the `kube-node-lease` namespace.\n - Webhooks that intercept [`Nodes`](https://kubernetes.io/docs/concepts/architecture/nodes/), [`TokenReviews`](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/), [`SubjectAccessReviews`](https://kubernetes.io/docs/reference/access-authn-authz/authorization/), and [`CertificateSigningRequests`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/) resources.\n\n Follow the instructions to\n [check webhooks that are considered unsafe](#unsafe-webhooks).\n\nWebhooks that have no available endpoints\n-----------------------------------------\n\nIf a webhook has no available endpoints, the Service that backs the webhook\nendpoint has one or more Pods which aren't running. To make the webhook\nendpoints available, follow the instructions to find and troubleshoot the Pods\nof the Service that is backing this webhook endpoint:\n\n1. Find the serving Pods for the Service associated with the webhook. Run the\n following command to describe the Service:\n\n kubectl describe svc \u003cvar translate=\"no\"\u003eSERVICE_NAME\u003c/var\u003e -n \u003cvar translate=\"no\"\u003eSERVICE_NAMESPACE\u003c/var\u003e\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSERVICE_NAME\u003c/var\u003e with the name of the Service.\n - \u003cvar translate=\"no\"\u003eSERVICE_NAMESPACE\u003c/var\u003e with the name of the namespace.\n\n If you can't find the Service name listed in the webhook, the unavailable\n endpoint might be caused by a mismatch between the name listed in the\n configuration and the actual name of the Service. To fix the endpoint\n availability, update the Service name in the webhook configuration to match\n the correct Service object.\n2. Inspect the serving Pods for this Service. Identify which Pods aren't\n running by listing the Deployment:\n\n kubectl get deployment -n \u003cvar translate=\"no\"\u003eSERVICE_NAMESPACE\u003c/var\u003e\n\n Or, run the following command to list the Pods: \n\n kubectl get pods -n \u003cvar translate=\"no\"\u003eSERVICE_NAMESPACE\u003c/var\u003e -o wide\n\n For any Pods that aren't running, inspect the Pod logs to see why the Pod\n isn't running.\n\nWebhooks that are considered unsafe\n-----------------------------------\n\nIf a webhook intercepts any resources in system-managed namespaces, we\nrecommend that you update the webhooks to avoid intercepting these resources.\n\n1. Inspect the webhook configuration. Run the following `kubectl` command to\n get the webhook configuration:\n\n kubectl get validatingwebhookconfigurations \u003cvar translate=\"no\"\u003eCONFIGURATION_NAME\u003c/var\u003e -o yaml\n\n Replace \u003cvar translate=\"no\"\u003eCONFIGURATION_NAME\u003c/var\u003e with the name of the\n webhook configuration.\n\n If this command doesn't return anything, run the command again, replacing\n `validatingwebhookconfigurations` with `mutatingwebhookconfigurations`.\n\n In the `webhooks` section of the output, one or more webhooks are listed.\n2. Edit the configuration, depending on the reason the webhook is considered\n unsafe:\n\n #### Exclude kube-system and kube-node-lease namespaces\n\n A webhook is considered unsafe if `scope` is `*`, or if scope is\n `Namespaced` and either of the following conditions are true:\n - The `operator` condition is `NotIn` and `values` omits `kube-system` and\n `kube-node-lease`, as in the following example:\n\n webhooks:\n - admissionReviewVersions:\n ...\n namespaceSelector:\n matchExpressions:\n - key: kubernetes.io/metadata.name\n operator: NotIn\n values:\n - blue-system # add 'kube-system' and 'kube-node-lease' if `NotIn`\n objectSelector: {}\n rules:\n - apiGroups:\n ...\n scope: '*' # 'Namespaced'\n sideEffects: None\n timeoutSeconds: 3\n\n Ensure that `scope` is set to `Namespaced`, not `*`, so that the webhook\n only operates in specific namespaces. Ensure that if `operator` is\n `NotIn`, `kube-system` and `kube-node-lease` are included in `values`.\n - The `operator` condition is `In` and `values` includes `kube-system` and\n `kube-node-lease`, as in the following example:\n\n namespaceSelector:\n matchExpressions:\n - key: kubernetes.io/metadata.name\n operator: In\n values:\n - blue-system\n - kube-system # remove as operator is `In`\n - kube-node-lease # remove as operator is `In`\n\n Ensure that `scope` is set to `Namespaced`, not `*`, so that the webhook\n only operates in specific namespaces. Ensure that if `operator` is `In`,\n `kube-system` and `kube-node-lease` are not included in `values`.\n\n #### Exclude matched resources\n\n A webhook is also considered unsafe if `nodes`, `tokenreviews`,\n `subjectaccessreviews`, or `certificatesigningrequests` are listed under\n resources, as in the following example: \n\n - admissionReviewVersions:\n ...\n resources:\n - 'pods' # keep, remove everything else\n - 'nodes'\n - 'tokenreviews'\n - 'subjectacessreviews'\n - 'certificatesigningrequests'\n scope: '*'\n sideEffects: None\n timeoutSeconds: 3\n\n Remove `nodes`, `tokenreviews`, `subjectaccessreviews`, and\n `certificatesigningrequests` from the resource section.\n\nWhat's next\n-----------\n\nIf you need additional assistance, reach out to\n\n[Cloud Customer Care](/support-hub).\n\nYou can also see\n[Getting support](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support) for more information about support resources, including the following:\n\n- [Requirements](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support#support_requirements) for opening a support case.\n- [Tools](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support#support_tools) to help you troubleshoot, such as logs and metrics.\n- Supported [components](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support#whats_supported), [versions](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support#version_support_policy), and [features](/kubernetes-engine/distributed-cloud/vmware/docs/getting-support#supported_features) of Google Distributed Cloud for VMware (software only)."]]