API proxy deployments fail with the following error messages.
Error Messages
If the TLS certificate of the
apigee-webhook-service.apigee-system.svc service has expired
or is not yet valid, the following error message will be shown on
apigee-watcher logs:
If there are duplicate certificate requests created for renewing the
apigee-serving-cert certificate, the
apigee-serving-cert certificate may not get renewed.
Make sure to include all of the settings shown, including --atomic
so that the action rolls back on failure.
Verify that the apigee-serving-cert certificate has been
created:
kubectl -n apigee-system get certificates apigee-serving-cert
Cause: Duplicate certificate requests were created for
renewing apigee-serving-cert
Diagnosis
Check cert-manager controller logs and see whether an error
message similar to the following has been returned.
List all cert-manager pods:
kubectl -n cert-manager get pods
An example output:
NAME READY STATUS RESTARTS AGE
cert-manager-66d9545484-772cr 1/1 Running 0 6d19h
cert-manager-cainjector-7d8b6bd6fb-fpz6r 1/1 Running 0 6d19h
cert-manager-webhook-669b96dcfd-6mnm2 1/1 Running 0 6d19h
Check cert-manager controller logs:
kubectl -n cert-manager logs cert-manager-66d9545484-772cr | grep "issuance is skipped until there are no more duplicates"
Example outputs:
1controller.go:163]cert-manager/certificates-readiness"msg"="re-queuing item due to error processing""error"="multiple CertificateRequests were found for the 'next' revision 3, issuance is skipped until there are no more duplicates""key"="apigee-system/apigee-serving-cert"
1controller.go:167]cert-manager/certificates-readiness"msg"="re-queuing item due to error processing""error"="multiple CertificateRequests were found for the 'next' revision 683, issuance is skipped until there are no more duplicates""key"="apigee/apigee-istiod"
If you see either of the messages shown above, the apigee-serving-cert and the apigee-istiod-cert certificates will not be renewed.
List all certificate requests in the apigee-system namespace or the apigee namespace
depending on the namespace printed in the log entries above and check to see if there are multiple certificate requests created for
renewing the same apigee-serving-cert or apigee-istiod-cert certificate revisions:
Verify that duplicated certificate requests have been deleted and only one
certificate request is available for the apigee-serving-cert
certificate in apigee-system namespace:
kubectl -n apigee-system get certificaterequests
Verify that the apigee-serving-cert certificate has been
renewed:
kubectl -n apigee-system get certificates apigee-serving-cert -o yaml
Check the health of the cert-manager pods in the
cert-manager namespace:
kubectl -n cert-manager get pods
If cert-manager pods are healthy, all
cert-manager pods should be ready (1/1) and in
Running state, otherwise, that could be the reason for this
issue:
NAME READY STATUS RESTARTS AGE
cert-manager-59cf78f685-mlkvx 1/1 Running 0 15d
cert-manager-cainjector-78cc865768-krjcp 1/1 Running 0 15d
cert-manager-webhook-77c4fb46b6-7g9g6 1/1 Running 0 15d
The cert-manager can fail for many reasons. Check the
cert-manager logs and identify the reason for the failure
and resolve them accordingly.
One known reason is that the cert-manager will fail if it
cannot communicate with the Kubernetes API. In this case, an error
message similar to following is displayed::
Refer to
Troubleshooting for additional cert-manager
troubleshooting information.
Must gather diagnostic information
If the problem persists even after following the above instructions, gather
the following diagnostic information, and then contact
Google Cloud Customer Care.
Google Cloud Project ID
Apigee hybrid organization
Apigee hybrid overrides.yaml file,
masking any sensitive information.
Kubernetes pod status in all namespaces:
kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt
Kubernetes cluster-info dump:
# generate kubernetes cluster-info dump
kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump
# zip kubernetes cluster-info dump
zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-28 UTC."],[[["\u003cp\u003eAPI proxy deployments in Apigee hybrid may fail due to an expired or invalid TLS certificate for the \u003ccode\u003eapigee-webhook-service.apigee-system.svc\u003c/code\u003e, leading to errors in \u003ccode\u003eapigee-watcher\u003c/code\u003e logs.\u003c/p\u003e\n"],["\u003cp\u003eThe issue can be caused by the \u003ccode\u003eapigee-serving-cert\u003c/code\u003e not being found, duplicate certificate requests, or an unhealthy \u003ccode\u003ecert-manager\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eResolutions involve updating the \u003ccode\u003eapigee-serving-cert\u003c/code\u003e using Helm, deleting duplicate certificate requests in the \u003ccode\u003eapigee-system\u003c/code\u003e namespace, and ensuring the \u003ccode\u003ecert-manager\u003c/code\u003e is healthy.\u003c/p\u003e\n"],["\u003cp\u003eIf problems persist, diagnostic information such as Google Cloud Project ID, Apigee hybrid organization details, \u003ccode\u003eoverrides.yaml\u003c/code\u003e file, Kubernetes pod status, and a Kubernetes \u003ccode\u003ecluster-info\u003c/code\u003e dump should be gathered for Google Cloud Customer Care.\u003c/p\u003e\n"]]],[],null,["*You're viewing **Apigee** and **Apigee hybrid** documentation.\nView [Apigee Edge](https://docs.apigee.com/api-platform/troubleshoot/support-d) documentation.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptoms\n\n\nAPI proxy deployments fail with the following error messages.\n\nError Messages\n\n\nIf the TLS certificate of the\n`apigee-webhook-service.apigee-system.svc` service has expired\nor is not yet valid, the following error message will be shown on\n`apigee-watcher` logs: \n\n```transact-sql\n{\"level\":\"error\",\"ts\":1687991930.7745812,\"caller\":\"watcher/watcher.go:60\",\n\"msg\":\"error during watch\",\"name\":\"ingress\",\"error\":\"INTERNAL: INTERNAL: failed\nto update ApigeeRoute [org-env]-group-84a6bb5, namespace apigee:\nInternal error occurred: failed calling webhook\n\\\"mapigeeroute.apigee.cloud.google.com\\\": Post\n\\\"https://apigee-webhook-service.apigee-system.svc:443/mutate-apigee-cloud-google-com-v1alpha1-apigeeroute?timeout=30s\\\":\nx509:\ncertificate has expired or is not yet valid: current time\n2023-06-28T22:38:50Z is after 2023-06-17T17:14:13Z, INTERNAL: failed to update\nApigeeRoute [org-env]-group-e7b3ff6, namespace apigee \n```\n\nPossible Causes\n\n| Cause | Description |\n|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [The apigee-serving-cert is not found](#not-found) | If the `apigee-serving-cert` is not found in the `apigee-system` namespace, this issue could occur. |\n| [Duplicate certificate requests were created for renewing `apigee-serving-cert`](#duplicate) | If there are duplicate certificate requests created for renewing the `apigee-serving-cert` certificate, the `apigee-serving-cert` certificate may not get renewed. |\n| [cert-manager is not healthy](#not-healthy) | If `cert-manager` is not healthy, the `apigee-serving-cert` certificate may not get renewed. |\n\nCause: The apigee-serving-cert is not found\n\nDiagnosis\n\n1.\n Check the availability of the `apigee-serving-cert` certificate\n in the `apigee-system` namespace:\n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert\n ```\n\n\n If this certificate is available, an output similar to following should\n be seen: \n\n ```text\n NAME READY SECRET AGE\n apigee-serving-cert True webhook-server-cert 2d10h\n ```\n2. If the apigee-serving-cert certificate is not found in the `apigee-system` namespace, that could be the reason for this issue.\n\nResolution\n\n1. Update the `apigee-serving-cert` using [Helm](/apigee/docs/hybrid/latest/helm-reference): \n\n ```\n helm upgrade ENV_NAME apigee-env/ \\\n --namespace APIGEE_NAMESPACE \\\n --set env=ENV_NAME \\\n --atomic \\\n -f OVERRIDES_FILE\n ```\n\n Make sure to include all of the settings shown, including `--atomic`\n so that the action rolls back on failure.\n2. Verify that the `apigee-serving-cert` certificate has been created: \n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert\n ```\n\nCause: Duplicate certificate requests were created for\nrenewing apigee-serving-cert\n\nDiagnosis\n\n1.\n Check `cert-manager` controller logs and see whether an error\n message similar to the following has been returned.\n\n List all `cert-manager` pods: \n\n ```\n kubectl -n cert-manager get pods\n ```\n\n An example output: \n\n ```text\n NAME READY STATUS RESTARTS AGE\n cert-manager-66d9545484-772cr 1/1 Running 0 6d19h\n cert-manager-cainjector-7d8b6bd6fb-fpz6r 1/1 Running 0 6d19h\n cert-manager-webhook-669b96dcfd-6mnm2 1/1 Running 0 6d19h\n ```\n\n Check `cert-manager` controller logs: \n\n ```\n kubectl -n cert-manager logs cert-manager-66d9545484-772cr | grep \"issuance is skipped until there are no more duplicates\"\n ```\n\n Example outputs: \n\n ```cbm-basic-v2\n 1 controller.go:163] cert-manager/certificates-readiness \"msg\"=\"re-queuing item due to error processing\" \"error\"=\"multiple CertificateRequests were found for the 'next' revision 3, issuance is skipped until there are no more duplicates\" \"key\"=\"apigee-system/apigee-serving-cert\"\n ``` \n\n ```cbm-basic-v2\n 1 controller.go:167] cert-manager/certificates-readiness \"msg\"=\"re-queuing item due to error processing\" \"error\"=\"multiple CertificateRequests were found for the 'next' revision 683, issuance is skipped until there are no more duplicates\" \"key\"=\"apigee/apigee-istiod\"\n ```\n\n\n If you see either of the messages shown above, the `apigee-serving-cert` and the `apigee-istiod-cert` certificates will not be renewed.\n2. List all certificate requests in the `apigee-system` namespace or the `apigee` namespace depending on the namespace printed in the log entries above and check to see if there are multiple certificate requests created for renewing the same `apigee-serving-cert` or `apigee-istiod-cert` certificate revisions: \n\n ```\n kubectl -n apigee-system get certificaterequests\n ```\n\n\nSee the `cert-manager` issue relevant to this problem at\n[cert-manager created multiple CertificateRequest objects with the same\ncertificate-revision](https://github.com/cert-manager/cert-manager/issues/4956).\n\nResolution **Note:** Execute the following commands for the `apigee` or `apigee-system` namespace depending on the namespace where the duplicate certificate requests were found.\n\n1. Delete all certificate requests in `apigee-system` namespace: \n\n ```\n kubectl -n apigee-system delete certificaterequests --all\n ```\n2. Verify that duplicated certificate requests have been deleted and only one certificate request is available for the `apigee-serving-cert` certificate in `apigee-system` namespace: \n\n ```\n kubectl -n apigee-system get certificaterequests\n ```\n3. Verify that the `apigee-serving-cert` certificate has been renewed: \n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert -o yaml\n ```\n\n An example output: \n\n ```actionscript-3\n apiVersion: cert-manager.io/v1\n kind: Certificate\n metadata:\n creationTimestamp: \"2023-06-26T13:25:10Z\"\n generation: 1\n name: apigee-serving-cert\n namespace: apigee-system\n resourceVersion: \"11053\"\n uid: e7718341-b3ca-4c93-a6d4-30cf70a33e2b\n spec:\n dnsNames:\n - apigee-webhook-service.apigee-system.svc\n - apigee-webhook-service.apigee-system.svc.cluster.local\n issuerRef:\n kind: Issuer\n name: apigee-selfsigned-issuer\n secretName: webhook-server-cert\n status:\n conditions:\n - lastTransitionTime: \"2023-06-26T13:25:11Z\"\n message: Certificate is up to date and has not expired\n observedGeneration: 1\n reason: Ready\n status: \"True\"\n type: Ready\n notAfter: \"2023-09-24T13:25:11Z\"\n notBefore: \"2023-06-26T13:25:11Z\"\n renewalTime: \"2023-08-25T13:25:11Z\"\n revision: 1\n ```\n\nCause: cert-manager is not healthy\n\nDiagnosis\n\n1. Check the health of the `cert-manager` pods in the `cert-manager` namespace: \n\n ```\n kubectl -n cert-manager get pods\n ```\n\n\n If `cert-manager` pods are healthy, all\n `cert-manager` pods should be ready `(1/1)` and in\n `Running` state, otherwise, that could be the reason for this\n issue: \n\n ```text\n NAME READY STATUS RESTARTS AGE\n cert-manager-59cf78f685-mlkvx 1/1 Running 0 15d\n cert-manager-cainjector-78cc865768-krjcp 1/1 Running 0 15d\n cert-manager-webhook-77c4fb46b6-7g9g6 1/1 Running 0 15d\n ```\n2.\n The `cert-manager` can fail for many reasons. Check the\n `cert-manager` logs and identify the reason for the failure\n and resolve them accordingly.\n\n\n One known reason is that the `cert-manager` will fail if it\n cannot communicate with the Kubernetes API. In this case, an error\n message similar to following is displayed:: \n\n ```transact-sql\n E0601 00:10:27.841516 1 leaderelection.go:330] error retrieving\n resource lock kube-system/cert-manager-controller: Get\n \"https://192.168.0.1:443/api/v1/namespaces/kube-system/configmaps/cert-manager-controller\":\n dial tcp 192.168.0.1:443: i/o timeout\n ```\n\nResolution\n\n1. Check the health of the Kubernetes cluster and fix any issues found. See [Troubleshooting Clusters](https://kubernetes.io/docs/tasks/debug/debug-cluster/).\n2. Refer to [Troubleshooting](https://cert-manager.io/docs/troubleshooting) for additional `cert-manager` troubleshooting information.\n\nMust gather diagnostic information\n\n\nIf the problem persists even after following the above instructions, gather\nthe following diagnostic information, and then contact\n[Google Cloud Customer Care](https://cloud.google.com/support-hub/).\n\n1. Google Cloud Project ID\n2. Apigee hybrid organization\n3. Apigee hybrid `overrides.yaml` file, masking any sensitive information.\n4. Kubernetes pod status in all namespaces: \n\n ```\n kubectl get pods -A \u003e kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt\n ```\n5. Kubernetes `cluster-info` dump: \n\n ```\n # generate kubernetes cluster-info dump\n kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump\n # zip kubernetes cluster-info dump\n zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*\n ```"]]
