Apigee ingress gateway pods show 1 of 2 containers running

Symptom

Your apigee-ingressgateway pods show only 1 of 2 containers running when you get the pod listing.

For example, when you run the following command:

kubectl -n apigee get pods -l app=apigee-ingressgateway

Output:

NAME                                                             READY   STATUS    RESTARTS   AGE
apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-shl9r         1/2     Running      0      6m48s
apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-vvzsf         1/2     Running      0      123m

Additionally, performing a describe command on one of the above pods will show the readiness probe failing with a 503 status code.

For example, in the Events section of the pod description, you might see the following message:

Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  76s (x32903 over 18h)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503

Possible cause

Cause: Apigee ingress is configured to listen on port 80

This issue is caused by the Apigee ingress gateway being configured to listen on port 80, which is no longer supported starting with Apigee Hybrid 1.9.

This can happen if you upgraded from an earlier version of Apigee Hybrid that allowed port 80, or if there's another misconfiguration that enabled it.

Diagnosis

1. Get a listing of your apigee-ingressgateway pods.

   Run the following command:

   kubectl -n apigee get pods -l app=apigee-ingressgateway

   Sample output

   NAME                                                              READY   STATUS    RESTARTS   AGE
   apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-shl9r         1/2     Running   0          6m48s
   apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-vvzsf         1/2     Running   0          123m
   

2. Describe one of the listed pods to check the events:

   kubectl -n apigee describe pod APIGEE_INGRESSGATEWAY_POD

   Where APIGEE_INGRESSGATEWAY_POD is an apigee-ingressgateway pod listed in the previous command output.

   Sample output:

   Events:
     Type     Reason     Age                    From     Message
     ----     ------     ----                   ----     -------
     Warning  Unhealthy  76s (x32903 over 18h)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
   

3. Get the logs for the APIGEE_INGRESSGATEWAY_POD pod.

   kubectl -n apigee logs APIGEE_INGRESSGATEWAY_POD

   You may see a log entry showing that port 80 failed to bind due to a permission denied error followed by a message that envoy is not ready.

   2025-09-11T06:16:45.457621Z     error   envoy config external/envoy/source/common/listener_manager/listener_manager_impl.cc:1186        listener '0.0.0.0_80' failed to bind or apply socket options: cannot bind '0.0.0.0:80': Permission denied
   ...
   2025-09-11T06:16:46.365818Z     warn    Envoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
   

4. Review the current apigeeroute configuration to find the one that has port 80 (HTTP) enabled.
   • Get a list of the defined apigee routes.

     kubectl -n apigee get apigeeroute

     Sample output:

     NAME                                  STATE     AGE
     myorg-hyb-dev-grp-000-33620d0         running   2d1h
     non-sni                               running   17s
     

   • Check each apigeeroute for where port 80 is defined.

     Run the following command for each apigeeroute listed in the previous command output:

     kubectl -n apigee get apigeeroute APIGEE_ROUTE_NAME -o yaml

     Where APIGEE_ROUTE_NAME is the name of an individual apigeeroute.

     Sample Command:

           kubectl -n apigee get apigeeroute non-sni -o yaml
           

     Sample output:

           apiVersion: apigee.cloud.google.com/v1alpha2
           kind: ApigeeRoute
           metadata:
             name: non-sni
             namespace: apigee
             resourceVersion: "240441468"
           spec:
             enableNonSniClient: true
             hostnames:
             - '*'
           ports:
             - number: 443
               protocol: HTTPS
               tls:
                 credentialName: myorg-hyb-dev-grp
                 minProtocolVersion: TLS_AUTO
                 mode: SIMPLE
             - number: 80
               protocol: HTTP
           selector:
             app: apigee-ingressgateway
           status:
             lastAppliedGeneration: 1
             state: running
           

     The non-sni apigeeroute shows that port 80 is enabled as part of this route.

Resolution

To resolve this issue, disable port 80 in the original apigeeroute yaml file by removing the following lines.

  - number: 80
    protocol: HTTP
      

If you do not have the original apigeeroute yaml file, you can follow the following steps:

1. Export the current configuration with the following command:

   kubectl -n apigee get apigeeroute APIGEE_ROUTE_NAME -o yaml > APIGEE_ROUTE_FILENAME.yaml

   Where APIGEE_ROUTE_NAME is the apigeeroute being updated to remove the port 80 configuration.

   Where APIGEE_ROUTE_FILENAME is the output file name containing the apigeeroute yaml content.

2. Edit the apigeeroute yaml file to remove the port 80 configuration.

   Open the APIGEE_ROUTE_FILENAME file in a text editor and remove the following lines:

           - number: 80
             protocol: HTTP
               

3. Replace the existing apigeeroute with the modified YAML file:

   kubectl -n apigee replace -f APIGEE_ROUTE_FILENAME.yaml
   

   Where APIGEE_ROUTE_FILENAME is the output file containing the updated apigeeroute configuration.

4. The apigee-ingressgateway pods should now run with 2/2 containers. If they do not, delete the pods to allow new ones to be created automatically.

           NAME                                                              READY   STATUS    RESTARTS   AGE
           apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-shl9r         2/2     Running   0          25h
           apigee-ingressgateway-ext-ingress-myorg-hyb-8f2c412-vvzsf         2/2     Running   0          26h
           

Must gather diagnostic information

• Overrides.yaml
• Output of the following commands:
  • kubectl -n apigee get pods -l app=apigee-ingressgateway

  • kubectl -n apigee logs APIGEE_INGRESSGATEWAY_POD

  • kubectl -n apigee get apigeeroutes

  • For each of the stated routes, run:

    kubectl -n apigee get apigeeroute APIGEE_ROUTE_NAME -o yaml

• As an alternative, you may run and provide the Apigee hybrid must-gather information.