Troubleshoot Network Connectivity Center

Learn about troubleshooting steps that you might find helpful if you run into problems using Network Connectivity Center, hybrid spokes, VPC spokes, and Private Service Connect connection propagation.

For known limitations, see Considerations on the Overview page.

Common issues with command syntax

When updating a spoke, you must specify the location that the spoke is in.

Attach resources to spokes

For detailed requirements, recommendations, and considerations for creating spokes and attaching resources to them, see Creating a spoke.

Routes aren't distributed between regions

If routes aren't being distributed properly between regions, verify that the VPC network's dynamic routing mode is set to global.

Data transfer traffic doesn't flow between two non-Google Cloud networks

In this context, a non-Google Cloud network refers to your on-premises data center, a branch office, or another cloud provider's network.

If data transfer traffic isn't flowing between two locations, confirm that the following resources are correctly configured and working:

Verify the functionality of HA VPN tunnels or VLAN attachments that are added as spokes.
Verify the routes programmed between the two locations.
Verify that ASN assignments are set up as described in ASN requirements for spoke resources.
Verify that each spoke has the site-to-site data transfer field set to true.

Duplicate route advertisements from BGP sessions

If there are duplicate route advertisements (some from BGP sessions participating in data transfer and some from sessions not participating in data transfer), then data transfer traffic might use ECMP to distribute traffic to all available next hops, even if those next hops aren't explicitly participating in data transfer.

Interconnect connection to an unsupported location (non-Google Cloud network)

For an example of how to configure route advertisements when one of your redundant Interconnect connections is to an unsupported location, see Optimal route advertisement for Network Connectivity Center.

Troubleshoot network connectivity using Connectivity Tests

Connectivity Tests is a diagnostics tool that lets you check connectivity between endpoints in your network. It analyzes your configuration and in some cases performs run-time verification.

To analyze network configurations, Connectivity Tests simulates the expected forwarding path of a packet through your Virtual Private Cloud (VPC) network, Cloud VPN tunnels, VLAN attachments, or router appliance instances.

You can use the Connectivity Tests configuration analysis to evaluate reachability between:

Two non-Google Cloud networks that are connected through Network Connectivity Center. In this context, a non-Google Cloud network is typically your on-premises data center or a branch office or another cloud provider's network.
Two VPC networks that are connected through a Network Connectivity Center hub.

Because Connectivity Tests doesn't have access to your on-premises network configuration, it can't verify the configuration of routes and firewall rules on your on-premises router. Thus, traffic from your on-premises network to your VPC network is always considered valid by the Connectivity Tests configuration analysis, and only configurations within Google Cloud are verified.

To run Connectivity Tests between two on-premises networks connected through Network Connectivity Center, see Running Connectivity Tests.

For more information, see the Connectivity Tests overview.

Troubleshoot hybrid spokes

The following issue can occur in Router appliance, VLAN attachment and VPN spokes.

Configured hub subnets aren't advertised to an on-premises network

If the configured hub subnets aren't advertised to the on-premises network (Preview), check the hub route table of the hybrid spoke's group for the following issues:

If the subnets are displayed in the hub route table, do the following:
- Make sure that the BGP route policy hasn't dropped the subnet to be advertised.
- Contact Google Cloud Support to diagnose the underlying issue.
If the subnets aren't in the hub route table, do the following:
- Check the include-export and exclude-export IP address ranges on the VPC spoke of the subnets. If the subnet is filtered by include-export ranges or exclude-export ranges, you can update the VPC spoke.

Troubleshoot Router appliance

The following issues and solutions are unique to Router appliance.

The documentation for troubleshooting Cloud Router also applies to Router appliance, along with the issues described in the following sections.

For more information, see the following:

BGP sessions fail to establish

If BGP sessions between Cloud Router and Router appliance fail to establish, check for the following issues:

Make sure that a VM acting as a router appliance instance is configured as part of a spoke in Network Connectivity Center. To configure a router appliance instance as part of a spoke, see Working with spokes.
Check your firewall settings to ensure that TCP port 179 is allowed. To configure firewall settings for Router appliance, see Create router appliance instances.
Make sure that neither Cloud Router nor your router appliance instance are using link-local addresses (that is, 169.254.x.x) to peer with each other. For guidance, see Recommendations for allocating IP addresses.
Make sure that Cloud Router has established two separate BGP sessions to your router appliance instance, one from each Cloud Router interface. The router appliance instance must advertise the same routes on both BGP sessions. If one of your BGP sessions goes down and your router appliance VM loses communication to Cloud Router, check the configuration of your Cloud Router interfaces. For more information, see Create router appliance instances.

Issues with internal IP addresses for BGP sessions on router appliance instances

If you discover issues related to your IP address configuration for router appliance instances, make sure that you have configured RFC 1918 internal IP addresses for BGP sessions between Cloud Router and the VMs acting as router appliance instances.

Router appliance instances don't use 169.254.x.x addresses for BGP sessions. Instead, they must use IP addresses in the same VPC subnet as Cloud Router. For more information, see Creating router appliance instances.

Troubleshoot VPC spokes

Permissions

If permission is denied when creating a spoke in a different project from the hub, check with the hub administrator to confirm that you have been granted the required networkconnectivity.groups.use permission on the hub.

VPC spoke creation failures

If your VPC spoke creation has failed, it could be due to one of the following reasons:

The VPC network is already peered with one or more of the existing spokes using VPC peering.
Subnets overlap with existing VPC spokes.
Subnets overlap with peers of existing VPC spokes.
You have exceeded the VPC spokes quota.
You have exceeded the routes per hub route table quota.
More than 16 export filters are specified.
Subnets in the VPC network are larger than one or more filters specified.

For quota-related errors, see Quotas and limits.

VPC spoke creation failures after deleting a spoke

After you delete a spoke, you must wait for the cool-down period of at least 10 minutes before you can create a new spoke for the same VPC network attached to a different hub. This cool-down period isn't needed if the VPC network is added as a spoke to the same hub.

Subnet creation failures in a VPC spoke

If you encounter subnet creation failure in a VPC spoke, it could be due to one of the following reasons:

There are overlapping subnets in other VPC spokes or peers of VPC spokes.
You have exceeded the routes per hub route table quota.
Subnets in the VPC network are larger than one or more filters specified.

The VPC spoke is created but dataplane connectivity is missing

If your VPC spoke is showing as created, but dataplane connectivity is missing, it could be due to one of the following reasons:

The spoke is in a different project from the hub and the hub administrator hasn't accepted the spoke proposal.
All subnets in the VPC network are filtered and excluded from connectivity.
Destination subnets are filtered by their corresponding VPC spoke filters.

The hub route table doesn't show some subnets in VPC spokes

If your hub route table doesn't show some of the subnets in the VPC spokes, it could be due to one of the following reasons:

The subnets are filtered using export filters.
The subnets were created within the last 5-10 minutes and the hub route table isn't refreshed yet.

Troubleshoot Private Service Connect connection propagation errors

Before investigating issues, familiarize yourself with the following pages:

VM in one spoke can't access the Private Service Connect connection in another spoke

If a VM in one VPC spoke can't access the Private Service Connect endpoint in another spoke, make sure that the following conditions are met:

The --export_psc field is enabled in the hub.

To check if the --export_psc field is enabled in the hub, use the gcloud network-connectivity hubs describe command.
gcloud
```
gcloud network-connectivity hubs describe HUB_NAME \
    --project=PROJECT_ID \
    --format='get(export_psc)'
```
Replace the following values:
- HUB_NAME: the name of the hub
- PROJECT_ID: the project ID where the hub resides
The IP address of the Private Service Connect endpoint isn't in any export exclude range of the hosting spoke. Private Service Connect endpoints that reside within the export exclude range aren't propagated.

To check the specified exclude export ranges of a spoke, use the gcloud network-connectivity spokes describe command.
gcloud
```
gcloud network-connectivity spokes describe SPOKE_NAME \
    --global \
    --project=PROJECT_ID \
    --format='get(linked_vpc_network.exclude_export_ranges)'
```
Replace the following values:
- SPOKE_NAME: the name of the spoke
- PROJECT_ID: the project ID where the spoke resides
You haven't exceeded the Network Connectivity Center usage quota.
The psc_connection_status of the endpoint is in the ACCEPTED state. For information about connection statuses, see Connection statuses.
The propagated connection status of the endpoint in the spoke containing the VM is READY. If no status is showing up, check for the reasons in the You can't see the status of some target spokes section on this page. You can check the propagated connection status by using the gcloud network-connectivity hubs query-status command:
```
  gcloud network-connectivity hubs query-status HUB_NAME\
      --filter='psc_propagation_status.source_spoke="SOURCE_SPOKE_NAME" AND psc_propagation_status.target_spoke="TARGET_SPOKE_NAME" AND psc_propagation_status.source_forwarding_rule="ENDPOINT_NAME"
```
Replace the following values:
- HUB_NAME: the name of the hub for which you want to check the connection propagation status
- SOURCE_SPOKE_NAME: the name of the source spoke
- TARGET_SPOKE_NAME: the name of the target spoke
- ENDPOINT_NAME: the name of the endpoint
For detailed information about how to use these flags, see the gcloud network-connectivity hubs query-status command page.
The host VPC network of the Private Service Connect endpoint (source) is a VPC spoke in the hub.
The IP address of the Private Service Connect endpoint is an RFC 1918 address.

You've exhausted the producer quota

If you see the PRODUCER_QUOTA_EXHAUSTED error message, you can ask the service producer to request a PSC_ILB_CONSUMER_FORWARDING_RULES_PER_PRODUCER_NETWORK quota increase. For details, see the Quotas per network section of the VPC documentation.

You've exceeded the producer's propagated connection limit

If you see an error about exceeding the propagated connection limit of a service attachment, you can ask the service producer to increase the limit. When a service producer updates the propagated connection limit, Google Cloud automatically checks if any pending propagated connections can be created.

The producer NAT IP address space is exhausted

If you see the PRODUCER_NAT_IP_SPACE_EXHAUSTED error message, it means that you might need to ask the service producer to add NAT subnets. For information about adding subnets, see Add or remove subnets from a published service.

You've exhausted the consumer quota

If you see a CONSUMER_QUOTA_EXHAUSTED error message, contact the owner of the consumer VPC network and request a PSC_PROPAGATED_CONNECTIONS_PER_VPC_NETWORK quota increase.

You can't see the status of some target spokes

If you don't see the status of some target spokes, it might be due to one of the following reasons:

The target spoke isn't a VPC spoke. Only VPC spokes, including VPC networks that are VPC spokes and also contain hybrid spokes, can receive propagated connections. A hybrid spoke in a routing VPC network that isn't also a VPC spoke can't receive propagated connections.
Network Connectivity Center only shows propagation status for each VPC spoke. If a VPC network is both a VPC spoke and contains hybrid spokes, check the propagation status of its VPC spoke to determine whether contained hybrid spokes have connectivity to propagated Private Service Connect endpoints.
If a VPC network is both the Private Service Connect producer VPC network and a VPC spoke on a hub that propagates its endpoints, Network Connectivity Center doesn't propagate its endpoints back to the producer VPC network itself.
The propagation is disallowed by the topology of the hub. For example, if the hub is configured to use star topology, the following conditions are also true:
- Private Service Connect endpoints in the center group are propagated to all other VPC spokes.
- Private Service Connect endpoints in the edge group are only propagated to VPC spokes in the center group.

Troubleshoot VPC spokes route exchange

VPC spokes and hybrid spokes are attached to the same hub but dataplane connectivity is missing

If the VPC spokes and hybrid spokes are attached to the same hub but dataplane connectivity is missing, it might be due to one of the following reasons:

The spoke is a cross-project spoke and the hub administrator hasn't accepted the spoke proposal.
All subnets in the VPC network are filtered and excluded from connectivity.
Auto propagation of VPC subnets isn't enabled or custom routes advertisement isn't set up.
The dynamic routes quota is exhausted.

The hub route table doesn't show some dynamic routes or shows spurious dynamic routes

The hub route table might not show dynamic routes correctly due to one of the following reasons:

BGP routes have been advertised or withdrawn within the last five to ten minutes and the hub route table isn't updated yet.
The dynamic routes quota is exhausted.

Hybrid spoke creation fails

If a hybrid spoke creation fails, it could be due to one of the following reasons:

The routing VPC network might be an existing VPC spoke to either the same hub or a different hub.
The routing VPC network might be implicitly associated with a different hub because of an existing hybrid spoke connected to that hub. Hybrid attachments from a VPC network can become spokes to only one hub.

VPC spoke creation fails

VPC spoke creation might fail if the VPC spoke is implicitly associated with a hub as a routing VPC network.

Error message

If you try to create a spoke and receive an error message that says An internal error occurred, to help troubleshoot the issue, contact Google Cloud Support.