Enable multi-cluster Gateways

This page shows you how to enable the multi-cluster GKE Gateway controller, a Google-hosted controller that provisions external and internal load balancers, for your GKE clusters. To learn how to use Gateway resources for container load balancing, see Deploying Gateways or Deploying multi-cluster Gateways.

The multi-cluster GKE Gateway Controller installs the following multi-cluster GatewayClasses in your clusters:

  • gke-l7-global-external-managed-mc for global external multi-cluster Gateways
  • gke-l7-regional-external-managed-mc for regional external multi-cluster Gateways
  • gke-l7-rilb-mc for regional internal multi-cluster Gateways
  • gke-l7-gxlb-mc for global external Classic multi-cluster Gateways

Learn more about the capabilities of the various GatewayClasses in GKE.

Pricing

All Compute Engine resources deployed through the Gateway controllers are charged against the project in which your GKE clusters reside. The single-cluster Gateway controller is offered at no additional charge as a part of GKE Standard and Autopilot pricing. Pricing for multi-cluster Gateways is described in the Multi Cluster Ingress and Gateway pricing page.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

GKE Gateway controller requirements

  • For Standard, GKE version 1.24 or later.
  • For Autopilot, GKE version 1.26 or later.
  • Google Cloud CLI version 407.0.0 or later.
  • The Gateway API is supported on VPC-native clusters only.
  • If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
  • Your cluster must have the HttpLoadBalancing add-on enabled.
  • If you are using Istio, you must upgrade Istio to one of the following versions:
    • 1.15.2 or later
    • 1.14.5 or later
    • 1.13.9 or later.
  • If you are using Shared VPC, then in the host project, you need to assign the Compute Network User role to the GKE Service account for the service project.

Multi-cluster Gateway requirements

In addition to the GKE Gateway controller requirements, for multi-cluster Gateway deployments, make sure you have performed the following tasks:

  • Enable the Gateway API on your config cluster.
  • Enable Workload Identity Federation for GKE on your clusters.
  • Complete the fleet general prerequisites for registering your clusters.

  • Enable the following multi-cluster Gateways required APIs in your project:

    • Cloud Service Mesh API
    • Multi-cluster Services API
    • Multi Cluster Ingress API

    Run this command to enable the required APIs if they are not already enabled:

    gcloud services enable \
  trafficdirector.googleapis.com \
  multiclusterservicediscovery.googleapis.com \
  multiclusteringress.googleapis.com \
  --project=PROJECT_ID

    Replace PROJECT_ID with the project ID where your GKE clusters are running.

Restrictions and Limitations

The same limitations and known issues for single-cluster Gateways also apply to multi-cluster Gateways.

In addition to the single-cluster Gateway limitations, the following limitations apply to multi-cluster Gateways:

  • Load balancing to backends in different regions is not supported with the regional internal GatewayClass gke-l7-rilb-mc. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.

  • A Service is not supported as a backendRefs with multi-cluster Gateway. Multi-cluster Gateway only supports ServiceImport as a valid backendRefs.

  • Cross-project load balancing is not supported. All clusters (config cluster and target clusters) attached to the same multi-cluster Gateway must be deployed in the same Shared VPC host project or service project. For more information on supported Shared VPC topologies with multi-cluster Gateway, see Use multi-cluster Gateway with Shared VPC.

  • Cross-VPC load balancing is not supported. All clusters (config cluster and target clusters) attached to the same multi-cluster Gateway must be deployed in the same VPC.

  • Multi-cluster Gateway depends on MCS for handling cross-cluster service discovery. As such, services multi-cluster Gateway exposes are subject to all the Multi-cluster Service requirements.

Quotas

GKE Gateway uses Cloud Load Balancing quotas to limit the number of resources that the Gateway controller can create to manage ingress traffic routed to GKE clusters.

Set up your environment for multi-cluster Gateways

It requires multiple GKE clusters to complete the examples in Deploying multi-cluster Gateways. All of the clusters are registered to the same fleet so that multi-cluster Gateways and Services can operate across them.

The following steps will deploy three GKE clusters across two different regions in your project:

  • us-west1-a/gke-west-1
  • us-west1-a/gke-west-2
  • us-east1-b/gke-east-1

This will create the following cluster topology:

The cluster topology which shows the relationship between the regions, fleet, and project.

These GKE clusters are used to demonstrate multi-region load balancing and blue-green, multi-cluster traffic splitting using external and internal Gateways.

Deploy clusters

In these steps you will deploy three GKE clusters into regions us-east1 and us-west1.

Clusters will be registered to your project's fleet. Grouping your GKE clusters together in a fleet allows them to be targeted by a multi-cluster Gateway.

  1. Create a GKE cluster in us-west1 named gke-west-1:

    gcloud container clusters create gke-west-1 \
    --gateway-api=standard \
    --zone=us-west1-a \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --cluster-version=VERSION \
    --enable-fleet \
    --project=PROJECT_ID

    Replace the following:

    • PROJECT_ID: the project ID where your GKE clusters are running.
    • VERSION: the GKE version, 1.24 or later.

  2. Create another GKE cluster in us-west1 (or the same region as the previous cluster) named gke-west-2:

    gcloud container clusters create gke-west-2 \
    --gateway-api=standard \
    --zone=us-west1-a \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --cluster-version=VERSION \
    --enable-fleet \
    --project=PROJECT_ID

  3. Create a GKE cluster in us-east1 (or a region that is different than the previous one) named gke-east-1

    gcloud container clusters create gke-east-1 \
    --gateway-api=standard \
    --zone=us-east1-b \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --cluster-version=VERSION \
    --enable-fleet \
    --project=PROJECT_ID

  4. Confirm that the clusters have been successfully registered to the fleet:

    gcloud container fleet memberships list --project=PROJECT_ID

    The output will be similar to the following:

    NAME        EXTERNAL_ID                           LOCATION
gke-east-1  45a80b37-4b00-49aa-a68b-b430fce1e3f0  us-east1
gke-west-2  ac7087a5-f5ee-401e-b430-57f3af141239  us-west1
gke-west-1  549efe3a-b18e-4eb9-8796-e50b7967cde2  us-west1

Configure cluster credentials

This step configures cluster credentials with memorable names. This makes it easier to switch between clusters when deploying resources across several clusters.

  1. Fetch the credentials for cluster gke-west-1, gke-west-2, and gke-east-1:

    gcloud container clusters get-credentials gke-west-1 --zone=us-west1-a --project=PROJECT_ID
gcloud container clusters get-credentials gke-west-2 --zone=us-west1-a --project=PROJECT_ID
gcloud container clusters get-credentials gke-east-1 --zone=us-east1-b --project=PROJECT_ID

    This stores the credentials locally so that you can use your kubectl client to access the cluster API servers. By default an auto-generated name is created for the credential.

  2. Rename the cluster contexts so they are easier to reference later:

    kubectl config rename-context gke_PROJECT_ID_us-west1-a_gke-west-1 gke-west-1
kubectl config rename-context gke_PROJECT_ID_us-west1-a_gke-west-2 gke-west-2
kubectl config rename-context gke_PROJECT_ID_us-east1-b_gke-east-1 gke-east-1

    Replace PROJECT_ID with the project ID where your clusters are deployed.

Enable multi-cluster Services in the fleet

  1. Enable multi-cluster Services in your fleet for the registered clusters. This enables the MCS controller for the three clusters that are registered to your fleet so that it can start listening to and exporting Services.

    gcloud container fleet multi-cluster-services enable \
    --project PROJECT_ID

  2. Grant Identity and Access Management (IAM) permissions required by the MCS controller:

    gcloud projects add-iam-policy-binding PROJECT_ID \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]" \
    --role "roles/compute.networkViewer" \
    --project=PROJECT_ID

    Replace PROJECT_ID with the project ID where your clusters are deployed.

  3. Confirm that MCS is enabled for the registered clusters. You will see the memberships for the three registered clusters. It may take several minutes for all of the clusters to show.

    gcloud container fleet multi-cluster-services describe --project=PROJECT_ID

    The output is similar to the following:

    createTime: '2023-10-12T06:14:33.466903587Z'
membershipStates:
  projects/441323991697/locations/us-east1/memberships/gke-east-1:
    state:
      code: OK
      description: Firewall successfully updated
      updateTime: '2023-10-12T06:15:28.395318091Z'
  projects/441323991697/locations/us-west1/memberships/gke-west-1:
    state:
      code: OK
      description: Firewall successfully updated
      updateTime: '2023-10-12T06:15:30.534594027Z'
  projects/441323991697/locations/us-west1/memberships/gke-west-2:
    state:
      code: OK
      description: Firewall successfully updated
      updateTime: '2023-10-12T06:15:29.110582109Z'
name: projects/pierre-louis-playground/locations/global/features/multiclusterservicediscovery
resourceState:
  state: ACTIVE
spec: {}
updateTime: '2023-10-12T06:15:31.027276757Z'

Enable multi-cluster Gateway in the fleet

The multi-cluster GKE Gateway controller governs the deployment of multi-cluster Gateways.

When enabling the multi-cluster Gateway controller, you must select your config cluster. The config cluster is the GKE cluster in which your Gateway resources (Gateway, Routes, Policies) are deployed. It is a central place that controls routing across your clusters. See Config cluster design to help you decide which cluster to choose as your config cluster.

  1. Enable multi-cluster Gateway and specify your config cluster in your fleet. Note that you can always update the config cluster at a later time. This example specifies gke-west-1 as the config cluster that will host the resources for multi-cluster Gateways.

    gcloud container fleet ingress enable \
    --config-membership=projects/PROJECT_ID/locations/us-west1/memberships/gke-west-1 \
    --project=PROJECT_ID

  2. Grant Identity and Access Management (IAM) permissions required by the multi-cluster Gateway controller:

    gcloud projects add-iam-policy-binding PROJECT_ID \
    --member "serviceAccount:service-PROJECT_NUMBER@gcp-sa-multiclusteringress.iam.gserviceaccount.com" \
    --role "roles/container.admin" \
    --project=PROJECT_ID

    Replace PROJECT_ID and PROJECT_NUMBER with the project ID and project number where your clusters are deployed.

  3. Confirm that the GKE Gateway controller is enabled for your fleet:

    gcloud container fleet ingress describe --project=PROJECT_ID

    The output is similar to the following:

    createTime: '2023-10-12T06:23:06.732858524Z'
membershipStates:
  projects/441323991697/locations/us-east1/memberships/gke-east-1:
    state:
      code: OK
      updateTime: '2023-10-12T06:30:08.815839024Z'
  projects/441323991697/locations/us-west1/memberships/gke-west-1:
    state:
      code: OK
      updateTime: '2023-10-12T06:30:08.815837031Z'
  projects/441323991697/locations/us-west1/memberships/gke-west-2:
    state:
      code: OK
      updateTime: '2023-10-12T06:30:08.815840985Z'
name: projects/pierre-louis-playground/locations/global/features/multiclusteringress
resourceState:
  state: ACTIVE
spec:
  multiclusteringress:
    configMembership: projects/pierre-louis-playground/locations/us-west1/memberships/gke-west-1
state:
  state:
    code: OK
    description: Ready to use
    updateTime: '2023-10-12T06:23:51.317464415Z'
updateTime: '2023-10-12T06:30:09.439319551Z'

  4. Confirm that the GatewayClasses exist in your config cluster:

    kubectl get gatewayclasses --context=gke-west-1

    The output is similar to the following:

    NAME                                  CONTROLLER                  ACCEPTED   AGE
gke-l7-global-external-managed        networking.gke.io/gateway   True       78m
gke-l7-global-external-managed-mc     networking.gke.io/gateway   True       4m22s
gke-l7-gxlb                           networking.gke.io/gateway   True       78m
gke-l7-gxlb-mc                        networking.gke.io/gateway   True       4m23s
gke-l7-regional-external-managed      networking.gke.io/gateway   True       78m
gke-l7-regional-external-managed-mc   networking.gke.io/gateway   True       4m22s
gke-l7-rilb                           networking.gke.io/gateway   True       78m
gke-l7-rilb-mc                        networking.gke.io/gateway   True       4m22s

    This output includes the GatewayClass gke-l7-global-external-managed-mc, gke-l7-regional-external-managed-mc, gke-l7-gxlb-mc for external multi-cluster Gateways and the GatewayClass gke-l7-rilb-mc for internal multi-cluster Gateways.

  5. Switch your kubectl context to the config cluster:

    kubectl config use-context gke-west-1

You are now ready to begin deploying multi-cluster Gateways in the config cluster.

Troubleshooting

This section shows you how to resolve issues related to the multi-cluster Gateway controller enablement.

GatewayClasses are not available in the config cluster

The following error might occur when you run the command kubectl get gatewayclasses:

error: the server doesn't have a resource type "gatewayclasses"

To resolve this issue, install the Gateway API on your cluster:

gcloud container clusters update CLUSTER_NAME \
    --gateway-api=standard \
    --region=COMPUTE_REGION

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.

Known Issues

  • There is a known issue with the multi-cluster Gateway controller where it may fail to start. To avoid this issue, enable the Gateway API on the config cluster and include the CRDs before the Fleet ingress feature is enabled.
  • Multi-cluster Gateways can leak load balancer resources under the following scenarios:
    • The Fleet ingress feature is updated with a new config cluster which does not have all Gateway resources which exist on the current config cluster.
    • The Fleet ingress feature is disabled while Gateway resources which reference a multi-cluster GatewayClass exist on the config cluster.
  • Multi-cluster Gateway runs as a global service. If the multi-cluster Gateway controller encounters a regional Fleet (Hub) control plane failure, it will respond by failing static and making no further load balancer changes until the region has been returned to service.

What's next