API requests fail with TARGET_CONNECT_HOST_NOT_REACHABLE error

You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Symptoms

API requests fail with TARGET_CONNECT_HOST_NOT_REACHABLE error.

Error Messages

If this issue occurs, the API requests will fail with HTTP 503 response status code and the following error:

{"fault":{"faultstring":
"Unable to resolve host invalid-target-host","detail":
{"errorcode":"protocol.http.NoResolvedHost","reason":
"TARGET_CONNECT_HOST_NOT_REACHABLE"}}}

Possible Causes

The following potential causes have been identified for the aforementioned symptom:

Cause: target server host specified is incorrect or has invalid characters

Diagnosis

1. Send an API request to the relevant API proxy:

   curl -ik https://dev.example.com/dns-peering-example
     HTTP/2 503
     content-type: application/json
     x-request-id: ***
     content-length: 169
     date: Thu, 02 Nov 2023 04:31:43 GMT
     via: 1.1 google
     alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

   and check the response message:

   {"fault":{"faultstring":
   "Unable to resolve host invalid-target-host","detail":
   {"errorcode":"protocol.http.NoResolvedHost","reason":
   "TARGET_CONNECT_HOST_NOT_REACHABLE"}}}

2. If the response contains the error reason TARGET_CONNECT_HOST_NOT_REACHABLE then, that is related to this reason.

Resolution

1. Check the API proxy definition and find the target hostname defined:
2. If the target hostname given is invalid or has invalid characters, fix it accordingly, create a new revision of the proxy, and deploy the proxy.

Cause: DNS peering is not configured

Diagnosis

1. Check whether the Apigee organization is peered with a VPC network by invoking the following Apigee API:

   TOKEN=$(gcloud auth print-access-token)

   curl -H "Authorization: Bearer $TOKEN" "https://apigee.googleapis.com/v1/organizations/$ORG" | jq .authorizedNetwork

   For example:

   TOKEN=$(gcloud auth print-access-token)

   curl -H "Authorization: Bearer $TOKEN" "https://apigee.googleapis.com/v1/organizations/example-org/" | jq .authorizedNetwork

   Response:

   "projects/example-org/global/networks/shared-vpc1"

   If the Apigee organization is not peered with a VPC and uses Private Service Connect (PSC), then, this scenario does not apply for this Apigee organization.

2. Verify with the API proxy developer from the customer side whether this target server domain name is internally configured. If not, this scenario does not apply.
3. Find the project ID and the network in which the target endpoint is hosted.
4. List DNS peerings created in the above network by executing the peered-dns-domains list command:

   gcloud services peered-dns-domains list --network=NETWORK --project=PROJECT-ID

   The above command could either return a blank result if no peered DNS domains are available, or list peered DNS domains similar to following:

   NAME                 DNS_SUFFIX
   customer-service     customer.service.internal.
   accounts-service     accounts.service.internal.

   If the above command does not list a DNS peering entry for the relevant DNS suffix, then, that could be the reason for this issue. Follow the instructions given in Resolution to resolve it.

Resolution

1. Make a note of the DNS suffix, project ID, and network in which the target endpoint is hosted.
2. Create a peered DNS domain for the DNS suffix using the peered-dns-domains create gcloud command. Note that the DNS suffix should contain a trailing dot at the end of the DNS suffix.

   gcloud services peered-dns-domains create NAME --network=NETWORK --dns-suffix=DNS-SUFFIX. --project=PROJECT-ID

   For example:

   gcloud services peered-dns-domains create orders-service --network="shared-vpc1" --dns-suffix="orders.service.internal." --project=service-project

   Response:

   Operation "operations/cpdd.p25-1064980322781-fafa5fe4-b5fe-487e-830d-fff0f9a6200d" finished successfully.

3. Now, send an API request to the API proxy endpoint and verify whether the API proxy could resolve the target server domain name and communicate with the target server:

   gcloud services peered-dns-domains create orders-service --network="shared-vpc1" --dns-suffix="orders.service.internal." --project=service-project

   Response:

   Operation "operations/cpdd.p25-1064980322781-fafa5fe4-b5fe-487e-830d-fff0f9a6200d" finished successfully

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information, and then contact Google Cloud Customer Care.

1. Google Cloud project ID
2. Apigee organization
3. API proxy and revision
4. Network in which the private domain is created
5. DNS suffix of the private domain
6. The complete output of the peered DNS domain creation command