Set up a hybrid mesh

This page explains how to set up a hybrid mesh for the following platforms:

  • Hybrid: GKE on Google Cloud and Google Distributed Cloud (software only) for VMware
  • Hybrid: GKE on Google Cloud and Google Distributed Cloud (software only) for bare metal

By following these instructions you set up two clusters, but you can extend this process to incorporate any number of clusters into your mesh.

Prerequisites

  • All clusters must be registered to the same fleet host project.
  • All GKE clusters must be in a shared VPC configuration on the same network.
  • The cluster's Kubernetes control plane address and the gateway address need to be reachable from every cluster in the mesh. The Google Cloud project in which GKE clusters are located should be allowed to create external load balancing types. We recommend that you use authorized networks and VPC firewall rules to restrict the access.
  • Private clusters, including GKE private clusters, are not supported. If you use On-Premises clusters including Google Distributed Cloud (software only) for VMware and Google Distributed Cloud (software only) for bare metal, the Kubernetes control plane address and the gateway address need to be reachable from pods in GKE clusters. We recommend that you use CloudVPN to connect the GKE cluster's subnet with the On-Premises cluster's network.
  • If you use Istio CA, use the same custom root certificate for all clusters.

Before you begin

You need access to the kubeconfig files for all the clusters that you are setting up in the mesh. For the GKE cluster, in order to create a new kubeconfig file for the cluster, you can export KUBECONFIG env with the complete path of file as value in your terminal and generate the kubeconfig entry.

Set up environment variables and placeholders

You need the following environment variables when you install the east-west gateway.

  1. Create an environment variable for the project number. In the following command, replace FLEET_PROJECT_ID with the project ID of the fleet host project.

    export PROJECT_NUMBER=$(gcloud projects describe FLEET_PROJECT_ID --format="value(projectNumber)")

  2. Create an environment variable for the mesh identifier.

    export MESH_ID="proj-${PROJECT_NUMBER}"

  3. Create environment variables for the network names.

    • GKE clusters default to the cluster network name:

      export NETWORK_1="PROJECT_ID-CLUSTER_NETWORK"

    • Other clusters use default:

      export NETWORK_2="default"

    Note that If you installed Cloud Service Mesh on other clusters with different values for --network_id, then you should pass the same values to value to NETWORK_2.

Install the east-west gateway

  1. Install a gateway in CLUSTER_1 (your GKE cluster) that is dedicated to east-west traffic to CLUSTER_2 (your on-premise cluster):

    asm/istio/expansion/gen-eastwest-gateway.sh \
    --mesh ${MESH_ID} \
    --network ${NETWORK_1}  \
    --revision asm-1246-9 | \
    ./istioctl --kubeconfig=PATH_TO_KUBECONFIG_1 install -y -f -

    Note that this gateway is public on the Internet by default. Production systems might require additional access restrictions, for example firewall rules, to prevent external attacks.

  2. Install a gateway in CLUSTER_2 that is dedicated to east-west traffic for CLUSTER_1.

    asm/istio/expansion/gen-eastwest-gateway.sh \
    --mesh ${MESH_ID} \
    --network ${NETWORK_2} \
    --revision asm-1246-9 | \
    ./istioctl --kubeconfig=PATH_TO_KUBECONFIG_2 install -y -f -

Expose services

Since the clusters are on separate networks, you need to expose all services (\*.local) on the east-west gateway in both clusters. While this gateway is public on the internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

Expose services via the east-west gateway for every cluster

    kubectl --kubeconfig=PATH_TO_KUBECONFIG_1 apply -n istio-system -f \
        asm/istio/expansion/expose-services.yaml
    kubectl --kubeconfig=PATH_TO_KUBECONFIG_2 apply -n istio-system -f \
        asm/istio/expansion/expose-services.yaml

Enable endpoint discovery

Run the asmcli create-mesh command to enable endpoint discovery. This example only shows two clusters, but you can run the command to enable endpoint discovery on additional clusters, subject to the GKE Hub service limit.

  ./asmcli create-mesh \
      FLEET_PROJECT_ID \
      PATH_TO_KUBECONFIG_1 \
      PATH_TO_KUBECONFIG_2

Verify multi-cluster connectivity

See Injecting sidecar proxies.