API proxy deployments fail with apigee-serving-cert is not found or expired

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptoms

API proxy deployments fail with the following error messages.

Error Messages

If the TLS certificate of the apigee-webhook-service.apigee-system.svc service has expired or is not yet valid, the following error message will be shown on apigee-watcher logs:

{"level":"error","ts":1687991930.7745812,"caller":"watcher/watcher.go:60",
"msg":"error during watch","name":"ingress","error":"INTERNAL: INTERNAL: failed
to update ApigeeRoute [org-env]-group-84a6bb5, namespace apigee:
Internal error occurred: failed calling webhook
\"mapigeeroute.apigee.cloud.google.com\": Post
\"https://apigee-webhook-service.apigee-system.svc:443/mutate-apigee-cloud-google-com-v1alpha1-apigeeroute?timeout=30s\":
x509:
certificate has expired or is not yet valid: current time
2023-06-28T22:38:50Z is after 2023-06-17T17:14:13Z, INTERNAL: failed to update
ApigeeRoute [org-env]-group-e7b3ff6, namespace apigee 

Possible Causes

Cause: The apigee-serving-cert is not found

Diagnosis

1. Check the availability of the apigee-serving-cert certificate in the apigee-system namespace:

   kubectl -n apigee-system get certificates apigee-serving-cert
   

   If this certificate is available, an output similar to following should be seen:

   NAME                  READY   SECRET                AGE
   apigee-serving-cert   True    webhook-server-cert   2d10h

2. If the apigee-serving-cert certificate is not found in the apigee-system namespace, that could be the reason for this issue.

Resolution

1. Update the apigee-serving-cert using Helm:

   helm upgrade ENV_NAME apigee-env/ \
     --namespace APIGEE_NAMESPACE \
     --set env=ENV_NAME \
     --atomic \
     -f OVERRIDES_FILE

   Make sure to include all of the settings shown, including --atomic so that the action rolls back on failure.

2. Verify that the apigee-serving-cert certificate has been created:

   kubectl -n apigee-system get certificates apigee-serving-cert

Cause: Duplicate certificate requests were created for renewing apigee-serving-cert

Diagnosis

1. Check cert-manager controller logs and see whether an error message similar to the following has been returned.

   List all cert-manager pods:

   kubectl -n cert-manager get pods

   An example output:

   NAME                                       READY   STATUS    RESTARTS        AGE
   cert-manager-66d9545484-772cr              1/1     Running   0               6d19h
   cert-manager-cainjector-7d8b6bd6fb-fpz6r   1/1     Running   0               6d19h
   cert-manager-webhook-669b96dcfd-6mnm2      1/1     Running   0               6d19h

   Check cert-manager controller logs:

   kubectl -n cert-manager logs cert-manager-66d9545484-772cr | grep "issuance is skipped until there are no more duplicates"

   Example outputs:

   1 controller.go:163] cert-manager/certificates-readiness "msg"="re-queuing item due to error processing" "error"="multiple CertificateRequests were found for the 'next' revision 3, issuance is skipped until there are no more duplicates" "key"="apigee-system/apigee-serving-cert"

   1 controller.go:167] cert-manager/certificates-readiness "msg"="re-queuing item due to error processing" "error"="multiple CertificateRequests were found for the 'next' revision 683, issuance is skipped until there are no more duplicates" "key"="apigee/apigee-istiod"

   If you see either of the messages shown above, the apigee-serving-cert and the apigee-istiod-cert certificates will not be renewed.

2. List all certificate requests in the apigee-system namespace or the apigee namespace depending on the namespace printed in the log entries above and check to see if there are multiple certificate requests created for renewing the same apigee-serving-cert or apigee-istiod-cert certificate revisions:

   kubectl -n apigee-system get certificaterequests

See the cert-manager issue relevant to this problem at cert-manager created multiple CertificateRequest objects with the same certificate-revision.

Resolution

1. Delete all certificate requests in apigee-system namespace:

   kubectl -n apigee-system delete certificaterequests --all

2. Verify that duplicated certificate requests have been deleted and only one certificate request is available for the apigee-serving-cert certificate in apigee-system namespace:

   kubectl -n apigee-system get certificaterequests

3. Verify that the apigee-serving-cert certificate has been renewed:

   kubectl -n apigee-system get certificates apigee-serving-cert -o yaml

   An example output:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     creationTimestamp: "2023-06-26T13:25:10Z"
     generation: 1
     name: apigee-serving-cert
     namespace: apigee-system
     resourceVersion: "11053"
     uid: e7718341-b3ca-4c93-a6d4-30cf70a33e2b
   spec:
     dnsNames:
     - apigee-webhook-service.apigee-system.svc
     - apigee-webhook-service.apigee-system.svc.cluster.local
     issuerRef:
       kind: Issuer
       name: apigee-selfsigned-issuer
     secretName: webhook-server-cert
   status:
     conditions:
     - lastTransitionTime: "2023-06-26T13:25:11Z"
       message: Certificate is up to date and has not expired
       observedGeneration: 1
       reason: Ready
       status: "True"
       type: Ready
     notAfter: "2023-09-24T13:25:11Z"
     notBefore: "2023-06-26T13:25:11Z"
     renewalTime: "2023-08-25T13:25:11Z"
     revision: 1

Cause: cert-manager is not healthy

Diagnosis

1. Check the health of the cert-manager pods in the cert-manager namespace:

   kubectl -n cert-manager get pods

   If cert-manager pods are healthy, all cert-manager pods should be ready (1/1) and in Running state, otherwise, that could be the reason for this issue:

   NAME                                       READY   STATUS    RESTARTS   AGE
   cert-manager-59cf78f685-mlkvx              1/1     Running   0          15d
   cert-manager-cainjector-78cc865768-krjcp   1/1     Running   0          15d
   cert-manager-webhook-77c4fb46b6-7g9g6      1/1     Running   0          15d

2. The cert-manager can fail for many reasons. Check the cert-manager logs and identify the reason for the failure and resolve them accordingly.

   One known reason is that the cert-manager will fail if it cannot communicate with the Kubernetes API. In this case, an error message similar to following is displayed::

   E0601 00:10:27.841516       1 leaderelection.go:330] error retrieving
   resource lock kube-system/cert-manager-controller: Get
   "https://192.168.0.1:443/api/v1/namespaces/kube-system/configmaps/cert-manager-controller":
   dial tcp 192.168.0.1:443: i/o timeout

Resolution

1. Check the health of the Kubernetes cluster and fix any issues found. See Troubleshooting Clusters.
2. Refer to Troubleshooting for additional cert-manager troubleshooting information.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information, and then contact Google Cloud Customer Care.

1. Google Cloud Project ID
2. Apigee hybrid organization
3. Apigee hybrid overrides.yaml file, masking any sensitive information.
4. Kubernetes pod status in all namespaces:

   kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt

5. Kubernetes cluster-info dump:

   # generate kubernetes cluster-info dump
   kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump
   # zip kubernetes cluster-info dump
   zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*