Deploying Gateways


This page describes how to deploy Kubernetes Gateway resources for load balancing ingress traffic to a single Google Kubernetes Engine (GKE) cluster.

For deploying Gateways to load balance ingress traffic across multiple clusters (or fleet), see Deploying Multi-Cluster Gateways.

For more specific Gateway configurations such as cross-namespace routing and HTTP traffic splitting, see the Gateway API user guides.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

GKE Gateway controller requirements

  • For Standard, GKE version 1.24 or later.
  • For Autopilot, GKE version 1.26 or later.
  • Google Cloud CLI version 407.0.0 or later.
  • The Gateway API is supported on VPC-native clusters only.
  • If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
  • Your cluster must have the HttpLoadBalancing add-on enabled.
  • If you are using Istio, you must upgrade Istio to one of the following versions:
    • 1.15.2 or later
    • 1.14.5 or later
    • 1.13.9 or later.
  • If you are using Shared VPC, then in the host project, you need to assign the Compute Network User role to the GKE Service account for the service project.

Restrictions and limitations

When using GKE Gateway, understand the following limitations and restrictions:

  • GKE GatewayClasses support different capabilities depending on the load balancer they use. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.

    Best practice:

    For optimal performance, limit the number of Gateways to a maximum of 100. Exceeding this limit can affect performance or result in increased latency.

  • You cannot use a FrontendConfig or a BackendConfig to configure a Gateway. You must use a Policy.

  • GKE Gateway behaves differently than Ingress, in that Gateway does not infer health check parameters. If your Service does not return 200 for requests to GET /, or you have other tuned pod readiness checks, you need to configure a HealthCheckPolicy for your service.

  • You cannot specify a port number directly in the hostname (for example, web.example.com:80) for traffic routing.

  • You can view the load balancer resources that GKE creates for Gateways in the Google Cloud console, but these resources don't reference the Gateway or GKE cluster they are attached to.

  • You can't automatically generate a Google-managed SSL certificate with Gateways but you can manually create and reference a Google-managed SSL certificate. For more information, see Secure a Gateway.
  • HTTPRoute is the only Route type supported. TCPRoutes, UDPRoutes, and TLSRoutes are not supported. To see a list of fields the GKE Gateway controller supports, see GatewayClass capabilities.

  • Custom request and response headers with Gateway or path redirects and URL rewrites with Gateway is only available on GKE version 1.27 or later.

  • For custom request and response headers with Gateway and path redirects and URL rewrites with Gateway, the GatewayClass gke-l7-gxlb is not supported.
  • When configuring HTTPRoute custom request and response headers, the following Google Cloud variables are not supported:

    • cdn_cache_id (Cloud CDN is not supported with GKE Gateway)
    • cdn_cache_status (Cloud CDN is not supported with GKE Gateway)
    • origin_request_header (CORS policies are not supported with GKE Gateway)
  • GKE Gateway does not support the Cloud CDN load balancing feature.

  • Mutual TLS custom headers are not supported (mTLS with GKE Gateway is not supported)

  • Google Cloud classic Application Load Balancer limitations apply to the GKE Gateway. In addition, you can't configure a custom Host response header in the backend service.

  • Path redirects and URL rewrites are mutually exclusive, you can't use both filters at the same time in the same rules.

  • Redirecting traffic to a different port is not supported with Cloud Load Balancing. To see the list of fields the GKE Gateway controller supports, see GatewayClass capabilities.

  • GKE Gateway does not support Wildcards, regular expressions, and dynamic URLs.

  • If you specify a Gateway with a regional external gateway class, the controller provisions an internal IP address instead of the external address. To learn how to use a named address with the regional external Application Load Balancer, see deploy a regional external Gateway.

  • Gateway utilizes Standalone NEGs for provisioning Network Endpoint Groups. To ensure that the Gateway controller properly reconciles the load balancer configuration, you cannot modify the cloud.google.com/neg annotation for a Service that is part of the Gateway.

  • GKE Gateway does not support referencing a Service that is also referenced by a GKE Ingress.

  • When a Gateway is configured to provision an IP address, changing the Gateway.spec.gatewayClass is not supported. To ensure that the Gateway controller properly reconciles the load balancer, delete the existing Gateway and re-deploy the manifest with the updated gatewayClass value.

  • The networking.gke.io/app-protocols annotation is not supported. Use the appProtocol field instead to achieve the same result.

  • If you use GKE Gateway with external-dns and the health state of the Gateway is unhealthy, by default, all DNS records associated with the Gateway are deleted from your DNS zones.

    Best practice:

    When running external-dns, set the policy=upsert-only flag. This configuration helps to prevent the deletion of existing DNS records.

  • If a port is removed from a Service that GKE Gateway references through a route, the Standalone NEG annotation on the Service you must also update the Standalone NEG controller on the Service to remove that port. If you don't, the NEG controller eventually stops syncing Pod endpoints for this Service. For details, see NEG Controller stops managing endpoints when port removed from Service.

Enable the Gateway API in your cluster

Before using Gateway resources in GKE, your cluster must have the Gateway API enabled.

Create a new cluster with the Gateway API enabled

GKE supports the Gateway API on Autopilot clusters starting with GKE version 1.26. If you create new Autopilot clusters on GKE 1.26 or later, the Gateway API is enabled by default. For existing clusters on GKE version 1.25 or earlier, the Gateway API is disabled by default.

If you create the cluster first without specifying a network (by using the --network flag), GKE creates the cluster in the default network. In that case, you must also create the proxy-only subnet in the default network. If you specify a network during cluster creation, ensure that you also create the proxy-only subnet in that same network.

Autopilot

Create a new GKE Autopilot cluster with the Gateway API enabled:

  gcloud container clusters create-auto CLUSTER_NAME \
      --location=CLUSTER_LOCATION \
      --release-channel=RELEASE_CHANNEL \
      --cluster-version=VERSION

Replace the following:

  • CLUSTER_NAME: the name of the cluster.
  • CLUSTER_LOCATION: the Compute Engine region or zone for the new cluster.
  • RELEASE_CHANNEL: the name of the release channel.
  • VERSION: the GKE version, which must be 1.26 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version of 1.26 or later.

When creating a cluster, you don't need to use the gcloud flag --gateway-api=standard flag.

Standard

With GKE Standard, the Gateway API is controlled by the --gateway-api flag. You can use the value standard when enabling and disabled when disabling.

Create a new VPC-native GKE cluster with the Gateway API enabled:

  gcloud container clusters create CLUSTER_NAME \
    --gateway-api=standard \
    --cluster-version=VERSION \
    --location=CLUSTER_LOCATION

Replace the following:

  • RELEASE_CHANNEL: the name of the release channel.
  • CLUSTER_NAME: the name of the cluster.
  • VERSION: the GKE version, which must be 1.24 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version of 1.24 or later.
  • CLUSTER_LOCATION: the Compute Engine region or zone for the new cluster.

The --gateway-api=standard flag instructs GKE to install the v1beta1 CRDs with the cluster.

Enable the Gateway API on an existing cluster

If you update an existing GKE Standard cluster to enable the Gateway API, make sure that the minimum requirements are met before proceeding with the update.

Ensure that your Autopilot cluster version is 1.26 or later and your Standard cluster version is 1.24 or later. To learn more about upgrading your Standard cluster version, see Standard cluster upgrades. To learn more about upgrading your Autopilot cluster version, see Autopilot cluster upgrades.

To enable the Gateway API on an existing GKE cluster (Autopilot or Standard), use the following command. When enabling Gateway on an existing cluster, it might take up to 45 minutes for the cluster to reconcile and install the CRDs.

gcloud container clusters update CLUSTER_NAME \
    --location=CLUSTER_LOCATION\
    --gateway-api=standard

Replace the following:

The --gateway-api=standard flag instructs GKE to install the v1beta1 CRDs with the cluster.

Verify your cluster

After creating or upgrading your cluster, the GKE Gateway controller automatically installs GatewayClasses. It might take a few minutes for the controller to recognize the CRDs and install the GatewayClasses.

  1. Confirm the Gateway API is enabled in the GKE control plane:

    gcloud container clusters describe CLUSTER_NAME \
      --location=CLUSTER_LOCATION \
      --format json
    

    The output is similar to the following. If this output is empty, re-run the cluster update command.

    "networkConfig": {
      ...
      "gatewayApiConfig": {
        "channel": "CHANNEL_STANDARD"
      },
      ...
    },
    
  2. Confirm the GatewayClasses are installed in your cluster:

    kubectl get gatewayclass
    

    The output is similar to the following:

    NAME                             CONTROLLER                  ACCEPTED   AGE
    gke-l7-global-external-managed   networking.gke.io/gateway   True       16h
    gke-l7-regional-external-managed networking.gke.io/gateway   True       16h
    gke-l7-gxlb                      networking.gke.io/gateway   True       16h
    gke-l7-rilb                      networking.gke.io/gateway   True       16h
    

To understand the capabilities of each GatewayClass, see GatewayClass capabilities.

Only single-cluster GatewayClasses are installed automatically. To install and use the multi-cluster GatewayClasses for internal and external multi-cluster load balancing, see Enabling multi-cluster Gateways.

Deploy an internal Gateway

An internal Gateway exposes applications that are only reachable from within the VPC or networks connected to the VPC.

Deploy a regional internal Gateway

The following example shows you how to deploy a regional internal Gateway that enables efficient and secure communication between services within a specific geographic region.

Configure a proxy-only subnet

You must configure a proxy-only subnet before you create a Gateway that uses an internal Application Load Balancer. Each region of a VPC in which you use internal Application Load Balancers must have a proxy-only subnet. This subnet provides internal IP addresses to the load balancer proxies.

  1. Create a proxy-only subnet:

    gcloud compute networks subnets create SUBNET_NAME \
        --purpose=REGIONAL_MANAGED_PROXY \
        --role=ACTIVE \
        --region=COMPUTE_REGION \
        --network=VPC_NETWORK_NAME \
        --range=CIDR_RANGE
    

    Replace the following:

    • SUBNET_NAME: the name of the proxy-only subnet.
    • COMPUTE_REGION: the region of the proxy-only subnet.
    • VPC_NETWORK_NAME: the name of the VPC network in which you create this proxy-only subnet. Ensure this is the same VPC network where your GKE cluster resides and where you deploy the Gateway. This is important for seamless communication between the load balancer and your backend services.
    • CIDR_RANGE: the primary IP address range of the subnet. You must use a subnet mask no longer than /26 so that at least 64 IP addresses are available for proxies in the region. The recommended subnet mask is /23.
  2. Verify your proxy-only subnet:

    gcloud compute networks subnets describe SUBNET_NAME \
        --region=COMPUTE_REGION
    

    The output is similar to the following:

    ...
    gatewayAddress: 10.1.1.1
    ipCidrRange: 10.1.1.0/24
    kind: compute#subnetwork
    name: proxy-subnet
    network: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/global/networks/default
    privateIpGoogleAccess: false
    privateIpv6GoogleAccess: DISABLE_GOOGLE_ACCESS
    purpose: REGIONAL_MANAGED_PROXY
    region: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION
    role: ACTIVE
    selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION/subnetworks/proxy-subnet
    state: READY
    

Create a Gateway

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. To learn more about the Gateway resource, see the Gateway resource description or the API specification.

In this case, the administrator of the GKE cluster wants to create a Gateway that can be used by different teams to expose their applications internally. The administrator deploys the Gateway, and application teams deploy their Routes independently and attach them to this Gateway.

  1. Save the following Gateway manifest to a file named gateway.yaml:

    kind: Gateway
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: internal-http
    spec:
      gatewayClassName: gke-l7-rilb
      listeners:
      - name: http
        protocol: HTTP
        port: 80
    

    This manifest includes the following fields:

    • gatewayClassName: gke-l7-rilb: specifies the GatewayClass that this Gateway is derived from. gke-l7-rilb corresponds to the internal Application Load Balancer.
    • port: 80: specifies that the Gateway exposes only port 80 for listening for HTTP traffic.
  2. Deploy the Gateway in your cluster:

    kubectl apply -f gateway.yaml
    
  3. Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.

    kubectl describe gateways.gateway.networking.k8s.io internal-http
    

    The output is similar to the following:

    Name:         internal-http
    Namespace:    default
    Spec:
      Gateway Class Name:  gke-l7-rilb
      Listeners:
        Allowed Routes:
          Kinds:
            Group:  gateway.networking.k8s.io
            Kind:   HTTPRoute
          Namespaces:
            From:  Same
        Name:      http
        Port:      80
        Protocol:  HTTP
    Status:
      Addresses:
        Type:   IPAddress
        Value:  192.168.1.14
      Conditions:
        Last Transition Time:  1970-01-01T00:00:00Z
        Message:               Waiting for controller
        Reason:                NotReconciled
        Status:                False
        Type:                  Scheduled
    Events:
      Type    Reason  Age                From                       Message
      ----    ------  ----               ----                       -------
      Normal  ADD     92s                networking.gke.io/gateway  test/internal-http
      Normal  UPDATE  45s (x3 over 91s)  networking.gke.io/gateway  test/internal-http
      Normal  SYNC    45s                networking.gke.io/gateway  SYNC on test/internal-http was a success
    

    At this point, there is a Gateway deployed in your cluster that has provisioned a load balancer and an IP address. The Gateway has no Routes, however, and so it doesn't know how it should send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404. Next, you deploy an application and Routes, which tell the Gateway how to get to application backends.

Deploy the demo applications

Application teams can deploy their applications and Routes independently from the deployment of Gateways. In some cases the application team might want to own the Gateway as well and deploy it themselves as a resource dedicated to their applications. See Route binding for different ownership models of Gateways and Routes. In this example however, the store team deploys their application and an accompanying HTTPRoute to expose their app through the internal-http Gateway created in the previous section.

The HTTPRoute resource has many configurable fields for traffic matching. For an explanation of HTTPRoute's fields, see the API specification.

  1. Deploy the store application (store-v1, store-v2, and store-german deployments) to your cluster:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml
    

    This creates three Deployments and three Services which are named store-v1, store-v2, and store-german.

  2. Validate that the application has deployed successfully:

    kubectl get pod
    

    The output is similar to the following after the application is running:

    NAME                        READY   STATUS    RESTARTS   AGE
    store-german-66dcb75977-5gr2n   1/1     Running   0          38s
    store-v1-65b47557df-jkjbm       1/1     Running   0          14m
    store-v2-6856f59f7f-sq889       1/1     Running   0          14m
    
  3. Validate that the Services have been deployed:

    kubectl get service
    

    The output shows a Service for each store Deployment:

    NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
    store-german   ClusterIP   10.48.3.183   <none>        8080/TCP   4s
    store-v1       ClusterIP   10.48.2.224   <none>        8080/TCP   5s
    store-v2       ClusterIP   10.48.4.48    <none>        8080/TCP   5s
    

Deploy the HTTPRoute

Route resources define protocol-specific rules for mapping traffic from a Gateway to Kubernetes backends. The HTTPRoute resource does HTTP and HTTPS traffic matching and filtering and is supported by all of the gke-l7 GatewayClasses.

In this section, you deploy an HTTPRoute, which programs the Gateway with the routing rules needed to reach your store application.

  1. Save the following HTTPRoute manifest to a file named store-route.yaml:

    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: store
    spec:
      parentRefs:
      - kind: Gateway
        name: internal-http
      hostnames:
      - "store.example.com"
      rules:
      - backendRefs:
        - name: store-v1
          port: 8080
      - matches:
        - headers:
          - name: env
            value: canary
        backendRefs:
        - name: store-v2
          port: 8080
      - matches:
        - path:
            value: /de
        backendRefs:
        - name: store-german
          port: 8080
    
  2. Deploy the HTTProute in your cluster:

    kubectl apply -f store-route.yaml
    

    The store HTTPRoute is bound to the internal-http Gateway by using the parentRefs property. These routing rules are configured on the underlying load balancer as in this diagram:

    The routing rules configured by the store HTTPRoute

    These routing rules process HTTP traffic in the following manner:

    • Traffic to store.example.com/de goes to the Service store-german.
    • Traffic to store.example.com with the HTTP header "env: canary" goes to the Service store-v2.
    • The remaining traffic to store.example.com goes to the Service store-v1.
  3. Verify that the HTTPRoute has been deployed:

    kubectl describe httproute store
    

    The output is similar to the following:

    Name:         store
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    API Version:  gateway.networking.k8s.io/v1beta1
    Kind:         HTTPRoute
    <...>
    Spec:
      Hostnames:
        store.example.com
      Parent Refs:
        Group:  gateway.networking.k8s.io
        Kind:   Gateway
        Name:   internal-http
      Rules:
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-v1
          Port:    8080
          Weight:  1
        Matches:
          Path:
            Type:   PathPrefix
            Value:  /
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-v2
          Port:    8080
          Weight:  1
        Matches:
          Headers:
            Name:   env
            Type:   Exact
            Value:  canary
          Path:
            Type:   PathPrefix
            Value:  /
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-german
          Port:    8080
          Weight:  1
        Matches:
          Path:
            Type:   PathPrefix
            Value:  /de
    Status:
      Parents:
        Conditions:
          Last Transition Time:  2022-11-01T04:18:52Z
          Message:
          Reason:                Accepted
          Status:                True
          Type:                  Accepted
          Last Transition Time:  2022-11-01T04:18:52Z
          Message:
          Reason:                ReconciliationSucceeded
          Status:                True
          Type:                  Reconciled
        Controller Name:         networking.gke.io/gateway
        Parent Ref:
          Group:  gateway.networking.k8s.io
          Kind:   Gateway
          Name:   internal-http
    Events:
      Type    Reason  Age                From                   Message
      ----    ------  ----               ----                   -------
      Normal  ADD     24m                sc-gateway-controller  default/store
      Normal  SYNC    16m (x4 over 23m)  sc-gateway-controller  Bind of HTTPRoute "default/store" to ParentRef {Group:       gateway.networking.k8s.io",
      <...>
    
  4. Verify that the HTTPRoute is bound to the Gateway:

    kubectl describe gateway
    

    The output is similar to the following:

    Name:         internal-http
    Namespace:    default
    Labels:       <none>
    <...>
    Status:
      Addresses:
        Type:   IPAddress
        Value:  10.128.15.203
      Conditions:
        Last Transition Time:  2022-11-01T03:47:01Z
        Message:
        Reason:                Scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2022-11-01T03:47:01Z
        Message:
        Reason:                Ready
        Status:                True
        Type:                  Ready
      Listeners:
        Attached Routes:  1
        Conditions:
          Last Transition Time:  2022-11-01T03:47:01Z
          Message:
          Reason:                Ready
          Status:                True
          Type:                  Ready
        Name:                    http
        Supported Kinds:
          Group:  gateway.networking.k8s.io
          Kind:   HTTPRoute
          <...>
    

Send traffic to your application

Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.

  1. Retrieve the IP address from the Gateway so that you can send traffic to your application:

    kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}"
    

    The output is an IP address.

  2. Send traffic to this IP address from shell on a virtual machine (VM) instance with connectivity to the cluster. You can create a VM for this purpose. This is necessary because the Gateway has an internal IP address and is only accessible from within your VPC network. Because the internal-http is a regional load balancer, the client shell must be within the same region as the GKE cluster.

    Make a request to store.example.com:

    curl https://store.example.com --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    Replace GATEWAY_IP_ADDRESS with the IP address from the previous step.

    The output from the demo app shows information about the location where the app is running:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "store-v1",
      "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
      "pod_name": "store-v1-84b47c7f58-pmgmk",
      "pod_name_emoji": "💇🏼‍♀️",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-10-25T13:31:17",
      "zone": "ZONE_NAME"
    }
    
  3. Test the path match by going to the German version of the store service at store.example.com/de:

    curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    The output confirms that the request was served by a store-german Pod:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "Gutentag!", 
      "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal",
      "pod_name": "store-german-5cb6474c55-lq5pl", 
      "pod_name_emoji": "🧞‍♀",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-10-25T13:35:37",
      "zone": "ZONE_NAME"
    }
    
  4. Finally, use the env: canary HTTP header to send traffic to the canary version of the store Service:

    curl -H "env: canary" https://store.example.com" --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    The output confirms that the request was served by a store-v2 Pod:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "store-v2", 
      "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
      "pod_name": "store-v2-5788476cbd-s9thb", 
      "pod_name_emoji": "🦰",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-10-25T13:38:26",
      "zone": "ZONE_NAME"
    }
    

Deploy an external Gateway

An external Gateway exposes applications that are reachable from the internet or networks outside of your VPC. The deployment is similar to an internal Gateway deployment except you must secure your applications because the Gateway is accessible to the public internet.

You have two options to create an external Gateway: a global external Gateway or a regional external Gateway.

A global external Gateway uses a Global IP address (or Anycast IP address) as the frontend of the Gateway advertised in all Google Cloud Compute regions. Clients sending traffic to this Anycast IP address are routed to the closest Google location where the IP is advertised. The global external Gateway is only available in the Premium Network Service tier.

A regional external Gateway uses a Regional IP as the frontend of the Gateway advertised only in the local Google Cloud Compute region where the regional external Gateway is deployed. Clients sending traffic to this regional IP address are routed by their local ISP and over the Internet before reaching the Google region where the IP is advertised. The regional external Gateway is only available in the Standard Network Service tier.

Deploy a global external Gateway

The following example shows you how to expose a store application with multiple certificates attached to the global external Gateway and grouped in a certificate map using Certificate Manager and an HTTPRoute.

Create a certificate map

Google recommends that you use Certificate Manager to manage certificates when you need 15 or more certificates per Gateway or you need to use wildcard certificates.

You can also secure your external Gateway using Kubernetes Secrets or Google-managed SSL certificates. For more information, see Gateway security.

In this section, you create certificates using Certificate Manager to secure the applications running on the cluster.

  1. Enable the Certificate Manager API:

    gcloud services enable certificatemanager.googleapis.com
    
  2. Create a certificate map:

    gcloud beta certificate-manager maps create store-example-com-map
    
  3. Load your Google-managed certificate and keys into a Certificate:

    gcloud beta certificate-manager certificates create store-example-com-cert \
        --certificate-file="CERTIFICATE_FILE" \
        --private-key-file="PRIVATE_KEY_FILE"
    

    Replace the following:

    • CERTIFICATE_FILE: the name of the new file that you choose. The file must have the extension .pem. For example, cert.pem.
    • PRIVATE_KEY_FILE: the name of your private key file.

    For more information, see Create a private key and certificate.

  4. Create a CertificateMapEntry which assigns the certificate to the certificate map:

    gcloud beta certificate-manager maps entries create store-example-com-map-entry \
        --map=store-example-com-map \
        --hostname=store.example.com \
        --certificates=store-example-com-cert
    

To learn how to secure a Gateway using other sources for certificates, such as Kubernetes Secrets or SSL certificates, see Secure a Gateway.

Create a Gateway

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it uses.

To learn more about the Gateway resource, see the Gateway resource description or the API specification.

In this section, you create a Gateway. Application teams can use the Gateway to expose their applications to the internet by deploying Routes independently and attaching them securely to the Gateway.

  1. Save the following manifest to a file named gateway.yaml:

    kind: Gateway
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: external-http
      annotations:
        networking.gke.io/certmap: store-example-com-map
    spec:
      gatewayClassName: gke-l7-global-external-managed
      listeners:
      - name: https
        protocol: HTTPS
        port: 443
    
    

    This manifest describes a Gateway with the following fields:

    • gatewayClassName: gke-l7-global-external-managed: specifies the GatewayClass for this Gateway. This gateway class uses a global external Application Load Balancer.
    • protocol: HTTPS and port: 443: specify that the Gateway exposes port 443 for HTTPS traffic. These fields enables TLS.
    • networking.gke.io/certmap: store-example-com-map: specifies the name of the certificate map in Certificate Manager.

    There is no TLS section because TLS is configured with Certificate Manager using the annotation networking.gke.io/certmap.

  2. Apply the manifest to your cluster:

    kubectl apply -f gateway.yaml
    

    It might take a few minutes for GKE to deploy the resources.

  3. Verify that the Gateway has deployed successfully:

    kubectl describe gateway
    

    The output is similar to the following:

    Name:         external-http
    Namespace:    default
    Labels:       <none>
    ...
    Spec:
      Gateway Class Name:  gke-l7-global-external-managed
      Listeners:
        Allowed Routes:
          Namespaces:
            From:  Same
        Name:      https
        Port:      443
        Protocol:  HTTPS
        Tls:
          Certificate Refs:
            Group:
            Kind:   Secret
            Name:   store-example-com
          Mode:     Terminate
     ...
    

    This output shows that the Gateway deployed in your cluster has a load balancer and a public IP address. The Gateway has no Routes, which means it can't send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404 response. In the next section, you deploy Routes, which instruct the Gateway to send traffic to backends.

Deploy the demo applications

Application teams can deploy their applications and Routes independently from the deployment of Gateways. In some cases the application team might want to own the Gateway as well and deploy it themselves as a resource dedicated to their applications. See Route binding for different ownership models of Gateways and Routes. In this example, the store team deploys their application and an accompanying HTTPRoute to expose their app through the external-http Gateway created in the previous section.

For more information about HTTPRoute fields, see the API specification.

  1. Deploy the sample application to your cluster:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml
    

    This sample application creates three Deployments and three Services which are named store-v1, store-v2, and store-german.

  2. Verify that the application has deployed successfully:

    kubectl get pod
    

    The output is similar to the following:

    NAME                            READY   STATUS    RESTARTS   AGE
    store-german-66dcb75977-5gr2n   1/1     Running   0          38s
    store-v1-65b47557df-jkjbm       1/1     Running   0          14m
    store-v2-6856f59f7f-sq889       1/1     Running   0          14m
    
  3. Verify that the Services have deployed successfully:

    kubectl get service
    

    The output is similar to the following:

    NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
    store-german   ClusterIP   10.48.3.183   <none>        8080/TCP   4s
    store-v1       ClusterIP   10.48.2.224   <none>        8080/TCP   5s
    store-v2       ClusterIP   10.48.4.48    <none>        8080/TCP   5s
    

Create an HTTPRoute

Route resources define protocol-specific rules for mapping traffic from a Gateway to Kubernetes backends. The HTTPRoute resource does HTTP and HTTPS traffic matching and filtering and is supported by all of the gke-l7-* GatewayClasses.

In this section, you deploy an HTTPRoute, which configures the Gateway with routing rules required to reach the sample application.

  1. Save the following manifest to a file named store-route-external.yaml:

    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: store-external
    spec:
      parentRefs:
      - kind: Gateway
        name: external-http
      hostnames:
      - "store.example.com"
      rules:
      - backendRefs:
        - name: store-v1
          port: 8080
      - matches:
        - headers:
          - name: env
            value: canary
        backendRefs:
        - name: store-v2
          port: 8080
      - matches:
        - path:
            value: /de
        backendRefs:
        - name: store-german
          port: 8080
    

    This manifest describes an HTTPRoute that references the external-http Gateway.

  2. Apply the manifest to your cluster:

    kubectl apply -f store-route-external.yaml
    

    The store HTTPRoute is bound to the external-http Gateway by using the parentRefs property. These following diagram shows the routing rules configured on the underlying load balancer:

    The routing rules configured by the store HTTPRoute

    The routing rules process HTTP traffic as follows:

    • Traffic to store.example.com/de routes to Service store-german.
    • Traffic to store.example.com with the HTTP header "env: canary" routes to Service store-v2.
    • The remaining traffic to store.example.com routes to Service store-v1.
  3. Verify that the HTTPRoute has been deployed:

    kubectl describe httproute store-external
    

    The output is similar to the following:

    Name:         store-external
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    API Version:  gateway.networking.k8s.io/v1beta1
    Kind:         HTTPRoute
    <...>
    Spec:
      Hostnames:
        store.example.com
      Parent Refs:
        Group:  gateway.networking.k8s.io
        Kind:   Gateway
        Name:   external-http
      Rules:
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-v1
          Port:    8080
          Weight:  1
        Matches:
          Path:
            Type:   PathPrefix
            Value:  /
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-v2
          Port:    8080
          Weight:  1
        Matches:
          Headers:
            Name:   env
            Type:   Exact
            Value:  canary
          Path:
            Type:   PathPrefix
            Value:  /
        Backend Refs:
          Group:
          Kind:    Service
          Name:    store-german
          Port:    8080
          Weight:  1
        Matches:
          Path:
            Type:   PathPrefix
            Value:  /de
    Status:
      Parents:
        Conditions:
          Last Transition Time:  2022-11-01T05:42:31Z
          Message:
          Reason:                Accepted
          Status:                True
          Type:                  Accepted
          Last Transition Time:  2022-11-01T05:43:18Z
          Message:
          Reason:                ReconciliationSucceeded
          Status:                True
          Type:                  Reconciled
        Controller Name:         networking.gke.io/gateway
        Parent Ref:
          Group:  gateway.networking.k8s.io
          Kind:   Gateway
          Name:   external-http
    Events:
      Type     Reason  Age    From                   Message
      ----     ------  ----   ----                   -------
      Normal   ADD     2m48s  sc-gateway-controller  default/store-external
      Normal  SYNC  61s (x3 over 2m27s)  sc-gateway-controller  Bind of HTTPRoute "default/store-external" to ParentRef Group:       "gateway.networking.k8s.io",
      ...
    
  4. Verify that the HTTPRoute is bound to the Gateway:

    kubectl describe gateway external-http
    

    The output is similar to the following:

    Name:         external-http
    Namespace:    default
    Labels:       <none>
    <...>
    Status:
      Addresses:
        Type:   IPAddress
        Value:  34.149.207.45
      Conditions:
        Last Transition Time:  2022-11-01T05:37:21Z
        Message:
        Reason:                Scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2022-11-01T05:43:18Z
        Message:
        Reason:                Ready
        Status:                True
        Type:                  Ready
      Listeners:
        Attached Routes:  1
        Conditions:
          Last Transition Time:  2022-11-01T05:43:18Z
          Message:
          Reason:                Ready
          Status:                True
          Type:                  Ready
        Name:                    https
        Supported Kinds:
          Group:  gateway.networking.k8s.io
          Kind:   HTTPRoute
          <...>
    

Send traffic to your application

Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.

  1. Get the IP address of the Gateway:

    kubectl get gateways.gateway.networking.k8s.io external-http -o=jsonpath="{.status.addresses[0].value}"
    

    The output is an IP address.

  2. Create a VM:

    gcloud cloud-shell ssh
    
  3. Send traffic to the Gateway IP address from the VM. You must set the host header manually because you do not own the example.com hostname.

    curl https://store.example.com --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    Replace GATEWAY_IP_ADDRESS with the IP address of the Gateway from the previous step.

    The output shows information from the demo app about the location where the app is running:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "store-v1",
      "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
      "pod_name": "store-v1-84b47c7f58-pmgmk",
      "pod_name_emoji": "💇🏼‍♀️",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-09-25T13:31:17",
      "zone": "us-central1-a"
    }
    
  4. Test the path match by going to the German version of the store service at store.example.com/de:

    curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    The output confirms that the request was served by a store-german Pod:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "Gutentag!",
      "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal",
      "pod_name": "store-german-5cb6474c55-lq5pl",
      "pod_name_emoji": "🧞‍♀",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-09-25T13:35:37",
      "zone": "us-central1-a"
    }
    
  5. Send traffic to the canary version of the store Service using the env: canary HTTP header:

    curl -H "env: canary" https://store.example.com"/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
    

    The output confirms that the request was served by a store-v2 Pod:

    {
      "cluster_name": "gke1",
      "host_header": "store.example.com",
      "metadata": "store-v2",
      "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
      "pod_name": "store-v2-5788476cbd-s9thb",
      "pod_name_emoji": "👩🏿",
      "project_id": "gateway-demo-243723",
      "timestamp": "2022-09-25T13:38:26",
      "zone": "us-central1-a"
    }
    

Deploy a regional external Gateway

The following example shows you how to expose a store application with multiple certificates attached to the regional external Gateway using self-managed certificates and an HTTPRoute.

Create a proxy subnet for your regional Gateway

You must configure a proxy-only subnet before you create a Gateway that uses a regional external Application Load Balancer. Each region of a VPC in which you use regional external Application Load Balancer must have an external_managed_proxy subnet. This subnet provides internal IP addresses to the load balancer proxies.

Create a certificate to secure your client traffic

You can use a certificate issued and validated by your certificate authority (CA) or create a self-signed certificate. For more information on how to create a certificate, see Store a certificate in a Kubernetes Secret.

CertificateMap or Google-managed SSL certificates are not supported with regional Gateways. Use self-managed regional SSL certificates or secrets to secure traffic between your clients and your regional Gateway. For more information about Certificate and Google Cloud load balancers, see Certificates and Google Cloud load balancers

Create a regional external HTTP(S) Gateway

  1. Create a regional static IP address for the external load balancer.

    gcloud compute addresses create IP_ADDRESS_NAME \
      --region=COMPUTE_REGION \
      --network-tier=STANDARD
    

    Replace the following:

    • IP_ADDRESS_NAME: the name of the new static IP address.
    • COMPUTE_REGION: The Compute Engine region where your cluster is running.
  2. Create a regional external Application Load Balancer Gateway using a self-managed certificate as follows and save the manifest as regional-gateway.yaml:

      kind: Gateway
      apiVersion: gateway.networking.k8s.io/v1beta1
      metadata:
        name: external-regional-http
      spec:
        gatewayClassName: gke-l7-regional-external-managed
        listeners:
        - name: https
          protocol: HTTPS
          port: 443
          tls:
            mode: Terminate
            certificateRefs:
            - name: store-example-com
        addresses:
        - type: NamedAddress
          value: IP_ADDRESS_NAME
    
  3. Apply the regional-gateway manifest:

      kubectl apply -f regional-gateway.yaml
    
  4. Verify your configuration.

      kubectl get gateway
    

    The output is similar to the following:

    NAME            CLASS                              ADDRESS         READY   AGE
    external-http   gke-l7-regional-external-managed   35.118.32.224   True    49s
    

    To get more details, use a describe command:

    kubectl describe gateway
    

    The output is similar to the following:

    Name:         external-regional-http
    Namespace:    default
    Labels:       <none>
    ...
    Spec:
      Gateway Class Name:  gke-l7-regional-external-managed
      Listeners:
        Allowed Routes:
          Namespaces:
            From:  Same
        Name:      https
        Port:      443
        Protocol:  HTTPS
        Tls:
          Certificate Refs:
            Group:
            Kind:   Secret
            Name:   store-example-com
          Mode:     Terminate
      ...
    

Deploy the demo application

You can deploy your applications and routes independently from the deployment of Gateways.

For more information on how to deploy the demo applications, see Deploy the demo applications.

Create an HTTPRoute

You must create an HTTPRoute to do HTTP and HTTPS traffic matching and filtering.

Send Traffic to your application

After you have deployed your application and created HTTPRoutes, you can pass traffic to your application.

For more information on how to send traffic to your application, see Send traffic to your application.

Use shared Gateways

The Gateway APIs use separate resources, Gateways and Route resources, to deploy load balancers and routing rules. This differs from Ingress, which combines everything in one resource. By splitting responsibility among resources, Gateway enables the load balancer and its routing rules to be deployed separately and to be deployed by different users or teams. This enables Gateways to become shared Gateways that attach with many different Routes that can be fully owned and managed by independent teams, even across different namespaces.

Deploy routes against a shared Gateway

This example builds on the internal-http Gateway deployed in Deploy an internal Gateway.

In this example, the site team deploys their application, Services, and an HTTPRoute to match traffic from the Gateway to those Services.

  1. Deploy the example application:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/site.yaml
    
  2. Save the following manifest to a file named site-route-internal.yaml:

    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: site-internal
    spec:
      parentRefs:
      - kind: Gateway
        name: internal-http
      hostnames:
      - "site.example.com"
      rules:
      - backendRefs:
        - name: site-v1
          port: 8080
    

    This manifest describes an HTTPRoute that matches all traffic for site.example.com and routes it to the site-v1 Service.

  3. Apply the manifest to your cluster:

    kubectl apply -f site-route-internal.yaml
    
  4. Verify that the HTTPRoute is attached to the Gateway:

    kubectl describe httproute.gateway.networking.k8s.io site-internal
    

    The output is similar to the following:

    Status:
      Parents:
        Conditions:
          Last Transition Time:  2023-01-09T15:05:43Z
          Message:
          Reason:                Accepted
          Status:                True
          Type:                  Accepted
          Last Transition Time:  2023-01-09T15:05:43Z
          Message:
          Reason:                ReconciliationSucceeded
          Status:                True
          Type:                  Reconciled
        Controller Name:         networking.gke.io/gateway
        Parent Ref:
          Group:  gateway.networking.k8s.io
          Kind:   Gateway
          Name:   internal-http
          ...
    

    If the Accepted condition for the Gateway is True, the HTTPRoute has successfully bound to the Gateway. To learn more about the Status field, see route status.

  5. Verify that traffic to the Gateway is routed correctly:

    curl -H "host: site.example.com" GATEWAY_IP_ADDRESS
    curl -H "host: store.example.com" GATEWAY_IP_ADDRESS
    

    Replace GATEWAY_IP_ADDRESS with the IP address of the Gateway.

    You must use a virtual machine (VM) in the same VPC as the Gateway.

    The output is similar to the following:

    {
      "cluster_name": "CLUSTER_NAME",
      "host_header": "site.example.com",
      "metadata": "site-v1",
      "pod_name": "site-v1-5d64fc4d7d-fz6f6",
      "pod_name_emoji": "👩🏼‍🍳",
      "project_id": "PROJECT_ID",
      "timestamp": "2022-11-02T19:07:01",
      "zone": "ZONE_NAME"
    }
    ...
    {
      "cluster_name": "CLUSTER_NAME",
      "host_header": "store.example.com",
      "metadata": "store-v1",
      "pod_name": "store-v1-6d8d58d78-vz8pn",
      "pod_name_emoji": "🧝🏻‍♂️",
      "project_id": "PROJECT_ID",
      "timestamp": "2022-11-02T19:07:01",
      "zone": "ZONE_NAME"
    }
    

Configure the Gateway default backend

All of the gke-l7-* GatewayClasses return HTTP 404 to unmatched traffic. You can configure the default backend using an explicit default Route that sends unmatched traffic to a user-provided Service.

Gateways are configured to handle error codes like 404 (Not Found) and 500 (Server Error), even without explicit backend definitions. The default behavior may vary between Gateway implementations. For greater control over error handling, consider configuring custom backends.

The following HTTPRoute is an example of how to customize the default backend. If you apply an HTTPRoute similar to the following, it takes precedence over the implicit default backend:

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: custom-default-backend
spec:
  parentRefs:
  - kind: Gateway
    name: my-internal-gateway
  rules:
  - backendRefs:
    - name: my-custom-default-backend-service
      port: 8080

This HTTPRoute matches all traffic from a particular Gateway. You can only have one such rule for each Gateway or else the rules conflict and precedence ordering applies.

You can use a default backend to prevent someone from creating a default route Backend that routes all Gateway traffic. An explicit HTTPRoute always takes precedence over new HTTPRoutes with conflicting routing rules.

Configure a static IP address for a Gateway

Every Gateway has an IP address it uses to listen for traffic. If you do not specify an IP address on the Gateway, then the Gateway controller automatically provides an IP address. You can also create a static IP address so that the IP address exists independent of the Gateway lifecycle.

After a Gateway is deployed, its IP address shows in the status field:

kind: Gateway
...
status:
  addresses:
    - value: 10.15.32.3

Depending on the GatewayClass, the IP address is allocated from the following subnets:

GatewayClasses Default IP Address Pool
  • gke-l7-rilb
  • gke-l7-rilb-mc
  • Regional private IP addresses from the primary node IPv4/IPv6 address range
  • gke-l7-regional-external-managed
  • gke-l7-regional-external-managed-mc
  • Regional public IP addresses from Google's regional external IPv4/IPv6 ranges
  • gke-l7-global-external-managed
  • gke-l7-global-external-managed-mc
  • gke-l7-gxlb
  • gke-l7-gxlb-mc
  • Global public IP addresses from Google's global external IPv4/IPv6 ranges

    The field addresses.NamedAddress lets you specify an IP address independently of the Gateway. You can create a static IP address resource prior to Gateway deployment and the resource is referenced by the NamedAddress. You can reuse the static IP address even if the Gateway is deleted.

    Use a named IP address

    You can configure an IPv4 or IPv6 address by specifying a NamedAddress. You must provision a static IP address before you create a Gateway.

    1. Create a static IP address resource:

      gcloud compute addresses create IP_ADDRESS_NAME \
          --purpose=SHARED_LOADBALANCER_VIP \
          --region=COMPUTE_REGION \
          --subnet=SUBNET \
          --project=PROJECT_ID
      

      Replace the following:

      • IP_ADDRESS_NAME: the name of the new static IP address
      • COMPUTE_REGION: for regional Gateways, the Compute Engine region where your cluster is running. This flag is not needed for global, external Gateways.
      • SUBNET: the subnet for the IP address. This flag is not needed for global, external Gateways.
      • PROJECT_ID: the project where your GKE cluster is running.
    2. Save the following manifest to a file named named-ip-gateway.yaml:

      kind: Gateway
      apiVersion: gateway.networking.k8s.io/v1beta1
      metadata:
        name: internal-http
      spec:
        gatewayClassName: gke-l7-rilb
        listeners:
        - name: http
          protocol: HTTP
          port: 80
        addresses:
        - type: NamedAddress
          value: IP_ADDRESS_NAME
      

      This manifest describes a Gateway that references the named IP address.

    3. Apply the manifest to your cluster:

      kubectl apply -f named-ip-gateway.yaml
      
    4. Verify your Gateway IP address:

      kubectl describe gateway internal-http
      

      The output is similar to the following:

      Name:         internal-http
      Namespace:    default
      Labels:       <none>
      ...
      Spec:
        Addresses:
          Type:              NamedAddress
          Value:             IP_ADDRESS_NAME
        Gateway Class Name:  gke-l7-rilb
        Listeners:
          Allowed Routes:
            Namespaces:
              From:  Same
          Name:      http
          Port:      80
          Protocol:  HTTP
      Status:
        Addresses:
          Type:   IPAddress
          Value:  10.15.32.103
      

    Configure HTTP-to-HTTPS redirects

    Cloud Load Balancing offers HTTP to HTTPS redirect functionality. An external Application Load Balancer redirects unencrypted HTTP requests to an HTTPS load balancer that uses the same IP address. When you create a Gateway with HTTP-to-HTTPS redirects enabled, both of these load balancers are created automatically. Requests to the external IP address of the Gateway on port 80 are automatically redirected to the same external IP address on port 443.

    By default, HTTP to HTTPS redirects are not defined on the Gateway.

    To redirect HTTP traffic to HTTPS, configure a Gateway to handle both HTTP and HTTPS traffic. If you disable either HTTP or HTTPS, the Gateway does not redirect traffic.

    The following example shows you how you can use HTTP-to-HTTPS redirect as a means to ensure that traffic from your clients going to your web applications is always being redirected to a secure page.

    HTTP-to-HTTPS redirects are not supported with the gke-l7-gxlb and gke-l7-gxlb-mc GatewayClasses. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.

    Redirect HTTP traffic from an infrastructure namespace

    In some cases, there isn't a clear distinction between the infrastructure or platform admin team and the application teams and preventing misuse of the Gateway can become a challenge.

    The following example further restricts the use of the HTTP listener to prevent unintentional use of non-secure protocol from the application teams. This example configures the Gateway to allow an HTTPRoute to use the HTTP listener only if the route is in a given namespace (http-redirect) while it opens the HTTPS listener to all namespaces. You can restrict the http-redirect namespace using Kubernetes RBAC so that application teams cannot create an HTTPRoute in this namespace by mistake.

    1. Create the namespace of a Gateway. Save the manifest as gateway-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: gateway-infra
      
    2. Apply the manifest:

      kubectl apply -f gateway-namespace.yaml
      
    3. Create the namespace of a Gateway and save the manifest as redirect-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: http-redirect
        labels:
          otherInfra: httpToHttps
      
      • Specific labels are set for this namespace.
    4. Apply the manifest:

      kubectl apply -f redirect-namespace.yaml
      
    5. To restrict the http listener usage, create a Gateway using the following manifest. Save the manifest as external-gateway.yaml:

      kind: Gateway
      apiVersion: gateway.networking.k8s.io/v1beta1
      metadata:
        name: external-http
        namespace: gateway-infra
      spec:
        gatewayClassName: gke-l7-global-external-managed
        listeners:
        - name: http
          protocol: HTTP
          port: 80
          allowedRoutes:
            kinds:
            - kind: HTTPRoute
            namespaces:
              from: Selector
              selector:
                matchLabels:
                  otherInfra: httpToHttps
        - name: https
          protocol: HTTPS
          port: 443
          allowedRoutes:
            kinds:
            - kind: HTTPRoute
            namespaces:
              from: All
          tls:
            mode: Terminate
            options:
              networking.gke.io/pre-shared-certs: store-example-com
        ```
      
      • The namespace field specifies that the Gateway is created in the gateway-infra namespace.

      • The namespaces field in the allowedRoutes section restricts the http listener to the namespace matching the label otherInfra: httpToHttps.

    6. Apply the manifest:

      kubectl apply -f external-gateway.yaml
      
    7. To force the HTTPS redirect, create a default HTTPRoute using the following manifest. Save the manifest as http-redirect.yaml:

      kind: HTTPRoute
      apiVersion: gateway.networking.k8s.io/v1beta1
      metadata:
        name: redirect
        namespace: http-redirect
      spec:
        parentRefs:
        - namespace: gateway-infra
          name: external-http
          sectionName: http
        rules:
        - filters:
          - type: RequestRedirect
            requestRedirect:
              scheme: https
      
      • The sectionName field instructs the Gateway to match only on the http listener. The RequestRedirect filter forces the redirection to the https listener.
    8. Apply the manifest:

      kubectl apply -f http-redirect.yaml
      
    9. Create a Service for an application using the following manifest. Save the manifest as service-deployment.yaml:

      apiVersion: v1
      kind: Service
      metadata:
        name: store-v1
      spec:
        selector:
          app: store
          version: v1
        ports:
        - port: 8080
          targetPort: 8080
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: store-v1
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: store
            version: v1
        template:
          metadata:
            labels:
              app: store
              version: v1
          spec:
            containers:
            - name: whereami
              image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.20
              ports:
              - containerPort: 8080
              env:
              - name: METADATA
                value: "store-v1"
      
    10. Apply the manifest:

      kubectl apply -f service-deployment.yaml
      
    11. Create an HTTPRoute for an application that only allows HTTPS using the following manifest. Save the manifest as http-route.yaml:

      kind: HTTPRoute
      apiVersion: gateway.networking.k8s.io/v1beta1
      metadata:
        name: store-external
        labels:
          gateway: external-http
      spec:
        parentRefs:
        - name: external-http
          namespace: gateway-infra
          sectionName: https
        hostnames:
        - "store.example.com"
        rules:
        - backendRefs:
          - name: store-v1
            port: 8080
      
    12. Apply the manifest:

      kubectl apply -f http-route.yaml
      

    Configure path redirects and URL rewrites

    Path redirects involve redirecting an incoming request from one URL path to another. Path redirects let you change the structure of the URL when you need to handle outdated or deprecated URLs.

    URL rewrites help modify the incoming URL before processing it on the server. It allows you to change the structure or format of the URL without actually changing the underlying content or file structure. URL rewriting is beneficial for creating user-friendly and SEO-friendly URLs that are easy to remember and understand. By default, path redirects and URL rewrites are not configured, you need to explicitly configure those redirects or rewrites using a filter in your HTTPRoute.

    GKE Gateway supports path redirects and URL rewrites. For more information, see HTTP path redirects and rewrites.

    Configure path redirects

    You can configure path redirects to either replace the entire path or only a prefix in the URL.

    Replace entire path

    1. To replace an entire path, configure a filter in an HTTPRoute that replaces any URL that contains the prefix /any-path in the URL path by the strict value /new-path.

    2. Create an HTTPRoute manifest as follows and name it as store.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
        kind: HTTPRoute
        metadata:
          name: store
        spec:
          parentRefs:
            - kind: Gateway
              name: external-http
          hostnames:
          - store.example.com
          rules:
          - matches:
            - path:
                type: PathPrefix
                value: /any-path
            filters:
            - type: RequestRedirect
              requestRedirect:
                path:
                  type: ReplaceFullPath
                  replaceFullPath: /new-path
                statusCode: 302
      

      For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/... should be redirected to a new location, https://store.example.com/new-path/ (strict).

    3. Apply the manifest:

      kubectl apply -f store.yaml
      

    This routing rule follows a strict redirection rule, which means that the browser does not attempt to cache the redirect, instead, redirects to the latest version.

    Replace a prefix only

    1. To replace a prefix only, configure a filter in an HTTPRoute that replaces any URL that contains the prefix /any-prefix in the URL path by the strict value /new-prefix.

    2. Create an HTTPRoute manifest as follows and name it as store.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
      kind: HTTPRoute
      metadata:
        name: store
      spec:
        parentRefs:
          - kind: Gateway
            name: external-http
        hostnames:
        - store.example.com
        rules:
        - matches:
            - path:
                type: PathPrefix
                value: /any-prefix
          filters:
          - type: RequestRedirect
            requestRedirect:
              path:
                type: ReplacePrefixMatch
                replacePrefixMatch: /new-prefix
              statusCode: 302
      

      For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/v1/... should be redirected to a new location, https://store.example.com/new-path/v1/... (only).

    3. Apply the manifest:

        kubectl apply -f store.yaml
      

    This routing rule follows the only redirection rule, which ensures that the browser always redirects you to the same intended page.

    Configure URL rewrites

    Set URL rewrites to change the way a URL appears to users. You can use URL rewrites to make URLs more user-friendly, to improve SEO, or to redirect users to a new page.

    Rewrite the entire hostname

    To rewrite the entire hostname:

    1. Configure a filter in an HTTPRoute that instructs the Gateway to replace the Host information in the request header from www.example.com to store.example.com before forwarding the request to the backend service.

    2. Create an HTTPRoute manifest as follows and name it as www.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
        kind: HTTPRoute
        metadata:
          name: www
        spec:
          parentRefs:
            - kind: Gateway
              name: external-http
          hostnames:
          - www.example.com
          rules:
          - filters:
            - type: URLRewrite
              urlRewrite:
                hostname: store.example.com
            backendRefs:
            - name: store-v1
              port: 8080
      

      For example, with the above configuration, any request to https://www.example.com is forwarded to the backend service with the Host: store.example.com header, instead of Host: www.example.com.

    3. Apply the manifest:

        kubectl apply -f www.yaml
      

    Rewrite using path modifiers

    You can combine rewrites with path modifiers to provide advanced URL and path modifications before relaying the request to the backend service.

    To rewrite using path modifiers:

    1. Configure a filter in an HTTPRoute that instructs the Gateway to replace the 'Host' information in the request header from www.example.comto store.example.com and replace the value /store by / before forwarding the request to the backend service.

    2. Create an HTTPRoute manifest as follows and name it as www.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
        kind: HTTPRoute
        metadata:
          name: www
        spec:
          parentRefs:
            - kind: Gateway
              name: external-http
          hostnames:
          - www.example.com
          rules:
          - matches:
            - path:
                type: PathPrefix
                value: /store
            filters:
            - type: URLRewrite
              urlRewrite:
                hostname: store.example.com
                path:
                  type: ReplacePrefixMatch
                  replacePrefixMatch: /de
            backendRefs:
            - name: store-german
              port: 8080
      

      For example, with the above configuration, any request to https://www.example.com/store/... is forwarded to the backend service with Host: store.example.com in the request header (instead of Host: www.example.com) and the /store is rewritten to /de.

    3. Apply the manifest:

      kubectl apply -f www.yaml
      

    Verify your configuration

    To verify that the filter was applied after creating your HTTPRoute with URL rewrite or path redirects filters, do the following:

    kubectl get httproute www -o yaml
    

    The output is similar to the following:

      apiVersion: gateway.networking.k8s.io/v1beta1
      kind: HTTPRoute
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"gateway.networking.k8s.io/v1beta1","kind":"HTTPRoute","metadata":{"annotations":{},"name":"www","namespace":"default"},"spec":{"hostnames":["www.example.com"],"parentRefs":[{"kind":"Gateway","name":"external-http"}],"rules":[{"backendRefs":[{"name":"store-german","port":8080}],"filters":[{"type":"URLRewrite","urlRewrite":{"hostname":"store.example.com","path":{"replacePrefixMatch":"/de","type":"ReplacePrefixMatch"}}}],"matches":[{"path":{"type":"PathPrefix","value":"/store"}}]}]}}
        creationTimestamp: "2023-06-22T01:00:42Z"
        generation: 3
        name: www
        namespace: default
        resourceVersion: "51268631"
        uid: e516493e-806d-44d6-ae0d-1c9ff25682cf
      spec:
        hostnames:
        - www.example.com
        parentRefs:
        - group: gateway.networking.k8s.io
          kind: Gateway
          name: external-http
        rules:
        - backendRefs:
          - group: ""
            kind: Service
            name: store-german
            port: 8080
            weight: 1
          filters:
          - type: URLRewrite
            urlRewrite:
              hostname: store.example.com
              path:
                replacePrefixMatch: /de
                type: ReplacePrefixMatch
          matches:
          - path:
              type: PathPrefix
              value: /store
      status:
        parents:
        - conditions:
          - lastTransitionTime: "2023-06-22T01:11:26Z"
            message: ""
            observedGeneration: 2
            reason: Accepted
            status: "True"
            type: Accepted
          - lastTransitionTime: "2023-06-22T01:11:26Z"
            message: ""
            observedGeneration: 2
            reason: ReconciliationSucceeded
            status: "True"
            type: Reconciled
          controllerName: networking.gke.io/gateway
          parentRef:
            group: gateway.networking.k8s.io
            kind: Gateway
            name: external-http
    
    

    To get more details, use the describe command:

    kubectl describe httproute
    

    Configure custom request and response headers

    Custom request and response headers let you specify additional headers to HTTP(S) requests and responses. Depending on the information detected by the load balancer, these headers can include the following information:

    • Latency to the client
    • Geographic location of the client's IP address
    • Parameters of the TLS connection

    By default, there are no custom headers added to the request sent/received to/from your backend services, you need to explicitly configure custom headers using a filter in your HTTPRoute.

    You can configure custom headers by adding a filter section in your HTTPRoute's rules as follows:

    Configure custom request headers

    Create a HTTPRoute manifest with a RequestHeaderModifier filter and save it as http-route-request.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
      kind: HTTPRoute
      metadata:
        name: store
      spec:
        <...>
        rules:
            filters:
              - type: RequestHeaderModifier
                requestHeaderModifier:
                  <...>
    

    Apply the manifest:

      kubectl apply -f http-route-request.yaml
    

    Configure custom response headers

    Create a HTTPRoute manifest with a ResponseHeaderModifier filter and save it as http-route-response.yaml:

    apiVersion: gateway.networking.k8s.io/v1beta1
    kind: HTTPRoute
    metadata:
      name: store
    spec:
      <...>
      rules:
          filters:
            - type: ResponseHeaderModifier
              responseHeaderModifier:
                <...>
    

    Apply the manifest:

      kubectl apply -f http-route-response.yaml
    

    You can add, set, and remove headers as described in the Gateway API implementation. You can configure your HTTPRoute with a custom header using Google Cloud supported variables.

    Example 1:

    To configure an HTTPRoute that adds client location information to the HTTP request before sending it to the backend service, create a HTTPRoute manifest and name it as external-http-request.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
      kind: HTTPRoute
      metadata:
        name: store
      spec:
        parentRefs:
          - kind: Gateway
            name: external-http
        hostnames:
        - store.example.com
        rules:
          - matches:
            - path:
                type: PathPrefix
                value: /fr
            filters:
              - type: RequestHeaderModifier
                requestHeaderModifier:
                  add:
                    - name: X-Client-Geo-Location
                      value: "{client_region},{client_city}"
            backendRefs:
              - name: store-french
                port: 8080
    

    For example, for clients located in Strasbourg, France, the Gateway adds a header as X-Client-Geo-Location:FR,Strasbourg.

    Example 2:

    To configure an HTTPRoute that adds a custom response header to support HTTP Strict Transport Security, create a HTTPRoute manifest and name it as external-http-response.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
      kind: HTTPRoute
      metadata:
        name: store
      spec:
        parentRefs:
          - kind: Gateway
            name: external-http
        hostnames:
        - store.example.com
        rules:
          - matches:
            - path:
                type: PathPrefix
                value: /de
            filters:
              - type: ResponseHeaderModifier
                responseHeaderModifier:
                  add:
                    - name: Strict-Transport-Security
                      value: max-age=63072000
            backendRefs:
              - name: store-german
                port: 8080
    

    Verify your configuration

    1. To verify your configuration after configuring custom request and response headers, do the following:

        kubectl get httproute
      

      The output is similar to the following:

        NAME    HOSTNAMES               AGE
        store   ["store.example.com"]   4d23h
      
    2. To get more details, use the describe command:

        kubectl describe httproute
      

      The output is similar to the following:

        Name:         store
        Namespace:    default
        Labels:       <none>
        Annotations:  <none>
        API Version:  gateway.networking.k8s.io/v1beta1
        Kind:         HTTPRoute
        Metadata:
          Creation Timestamp:  2023-05-27T00:51:01Z
          Generation:          5
          Resource Version:    25418887
          UID:                 2e07a1b8-420b-41b4-acd1-cecbfcd39f42
        Spec:
          Hostnames:
            store.example.com
          Parent Refs:
            Group:  gateway.networking.k8s.io
            Kind:   Gateway
            Name:   external-http
          Rules:
            Backend Refs:
              Group:
              Kind:    Service
              Name:    store-v1
              Port:    8080
              Weight:  1
            Matches:
              Path:
                Type:   PathPrefix
                Value:  /
            Backend Refs:
              Group:
              Kind:    Service
              Name:    store-v2
              Port:    8080
              Weight:  1
            Matches:
              Headers:
                Name:   env
                Type:   Exact
                Value:  canary
              Path:
                Type:   PathPrefix
                Value:  /
            Backend Refs:
              Group:
              Kind:    Service
              Name:    store-german
              Port:    8080
              Weight:  1
            Filters:
              Request Header Modifier:
                Add:
                  Name:   X-Client-Geo-Location
                  Value:  {client_region},{client_city}
              Type:       RequestHeaderModifier
            Matches:
              Path:
                Type:   PathPrefix
                Value:  /de
        Status:
          <...>
      

    Route status

    HTTPRoute resources emit conditions and events to help users understand if a HTTPRoute has successfully bound with one or more Gateways or if it was rejected.

    HTTPRoute conditions

    HTTPRoute conditions indicate the status of the Route and the Gateways it is bound to. Because a Route can be bound to multiple Gateways, this is a list of Gateways and the individual conditions between the Route and each Gateway.

    • Accepted=True indicates that the HTTPRoute is successfully bound to a Gateway.
    • Accepted=False indicates that the HTTPRoute has been rejected from binding with this Gateway.

    If there are no Gateways listed under the Gateway bindings heading, then your HTTPRoute labels and Gateway label selectors might not match. This can occur if your Route is not being selected by any Gateways.

    HTTPRoute events

    HTTPRoute events provide details about the status of the HTTPRoute. Events are grouped by the following reasons:

    • ADD events are triggered by a resource being added.
    • UPDATE events are triggered by a resource being updated.
    • SYNC events are triggered by periodic reconciliation.

    Route merging, precedence, and validation

    Route precedence

    The Gateway API defines strict precedence rules for how traffic is matched by Routes that have overlapping routing rules. The precedence between two overlapping HTTPRoutes is as follows:

    1. Hostname merge: The longest/most specific hostname match.
    2. Path merge: The longest/most specific path match.
    3. Header merge: The largest number of HTTP headers that match.
    4. Conflict: If the previous three rules don't establish precedence, then precedence goes to the HTTPRoute resource with the oldest timestamp.

    Route merging

    For gke-l7 GatewayClasses, all HTTPRoutes for a given Gateway are merged into the same URL map resource. How the HTTPRoutes are merged together depends on the type of overlap between HTTPRoutes. The HTTPRoute from the earlier example can be split into three separate HTTPRoutes to illustrate route merging and precedence:

    1. Route merge: All three HTTPRoutes attach with the same internal-http Gateway, so they are merged together.
    2. Hostname merge: All three Routes match for store.example.com, so their hostname rules are merged.
    3. Path merge: store-german-route has a more specific path /de, so this is not merged further. store-v1-route and store-v2-route both match on the same /* path as well, so they are merged on the path.
    4. Header merge: store-v2-route has a more specific set of HTTP header matches than store-v1-route, so they are not merged further.
    5. Conflict: Because the Routes are able to be merged on hostname, path, and headers, there are no conflicts, and all of the routing rules are applied to traffic.

    The single HTTPRoute used in the earlier example are equivalent to these three separate routes:

    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: store-v1-route
    spec:
      parentRefs:
      - kind: Gateway
        name: internal-http
      hostnames:
      - "store.example.com"
      rules:
      - backendRefs:
        - kind: Service
          name: store-v1
          port: 8080
    ---
    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: store-v2-route
    spec:
      parentRefs:
      - kind: Gateway
        name: internal-http
      hostnames:
      - "store.example.com"
      rules:
      - matches:
        - headers:
          - type: Exact
            name: env
            value: canary
        backendRefs:
        - kind: Service
          name: store-v2
          port: 8080
    ---
    kind: HTTPRoute
    apiVersion: gateway.networking.k8s.io/v1beta1
    metadata:
      name: store-german-route
    spec:
      parentRefs:
      - kind: Gateway
        name: internal-http
      hostnames:
      - "store.example.com"
      rules:
      - matches:
        - path:
            type: PathPrefix
            value: /de
        backendRefs:
        - kind: Service
          name: store-german
          port: 8080
    

    Kubernetes Gateways and Istio Gateways

    Note that the Kubernetes Gateway API and the Istio API both have a resource named Gateway. While they perform similar functions, they are not the same resource. If you are using Istio and the Gateway API in the same Kubernetes cluster, these names overlap when using kubectl on the command line. kubectl get gateway might return the Kubernetes Gateway resources and not the Istio Gateway resources or vice versa.

    $ kubectl api-resources
    NAME       SHORTNAMES   APIGROUP                       NAMESPACED   KIND
    gateways   gw           networking.istio.io/v1beta1    true         Gateway
    gateways   gtw          networking.k8s.io/v1beta1      true         Gateway
    

    If you are using Istio and upgrade to GKE 1.20 and later it is recommended to start using the Gateway resource shortname or specify the API group. The shortname for a Kubernetes Gateway is gtw and the shortname for an Istio Gateway is gw. The following commands return the Kubernetes Gateway and Istio Gateway resources respectively.

    # Kubernetes Gateway
    $ kubectl get gtw
    NAME                        CLASS
    multi-cluster-gateway       gke-l7-global-external-managed-mc
    
    $ kubectl get gateway.networking.x-k8s.io
    NAME                        CLASS
    multi-cluster-gateway       gke-l7-global-external-managed-mc
    
    # Istio Gateway
    $ kubectl get gw
    NAME               AGE
    bookinfo-gateway   64m
    
    $ kubectl get gateway.networking.istio.io
    NAME               AGE
    bookinfo-gateway   64m
    

    Troubleshooting

    Proxy-only subnet missing in the region

    Symptom:

    The following issue might occur when you create a regional Gateway (internal or external):

    generic::invalid_argument: error ensuring load balancer: Insert: Invalid value for field 'resource.target': 'regions/[REGION_NAME]/targetHttpProxies/gkegw-x5vt-default-internal-http-[ID]'. A reserved managed proxy subnetwork with purpose REGIONAL_MANAGED_PROXY is required.
    

    Reason:

    This error message indicates that no proxy-only subnet exists in the region for your Gateway.

    Workaround:

    To resolve this issue, configure a proxy-only subnet.

    Proxy-only subnet already exists in the region with the wrong purpose

    Symptom:

    The following issue might occur when you create a proxy-only subnet for your regional Gateway (internal or external):

    ERROR: (gcloud.compute.networks.subnets.create) Could not fetch resource:
     - The resource 'projects/[PROJECT_NAME]/regions/[REGION_NAME]/subnetworks/[PROXY_ONLY_SUBNET_NAME]' already exists
    

    Reason:

    This error message indicates that you attempted to create a regional proxy-only subnet in a region that already has a proxy-only subnet.

    Workaround:

    To resolve this issue, use the following steps:

    1. Check that a proxy-only subnet already exists in the region and verify that it has the correct purpose:

      1. List your subnets to find which one is the proxy-only subnet in the region:

        gcloud compute networks subnets list --regions=COMPUTE_REGION
        

        Replace COMPUTE_REGION with the Compute Engine region where you want to create your regional Gateway.

      2. Describe your proxy-only subnet in the region to find its purpose:

        gcloud compute networks subnets describe PROXY_ONLY_SUBNET \
            --region COMPUTE_REGION | grep -E 'name|purpose'
        

        Replace PROXY_ONLY_SUBNET with the proxy-only subnet.

      GKE Gateway only supports REGIONAL_MANAGED_PROXY proxy-only subnets for regional Gateways (internal or regional).

    2. If the existing proxy-only subnet in the region was created with an INTERNAL_HTTPS_LOAD_BALANCER purpose, migrate its purpose to REGIONAL_MANAGED_PROXY.

    No healthy upstream

    Symptom:

    The following issue might occur when you create a Gateway but cannot access the backend services (503 response code):

    no healthy upstream
    

    Reason:

    This error message indicates that the health check prober cannot find healthy backend services. It is possible that your backend services are healthy but you might need to customize the health checks.

    Workaround:

    To resolve this issue, customize your health check based on your application's requirements (for example, /health) using a HealthCheckPolicy.

    What's next