User-managed firewall rules for GKE LoadBalancer Services

This page describes how to disable the ingress allow VPC firewall rules that GKE creates for LoadBalancer Services.

Disabling these automatically created firewall rules for LoadBalancer Services can be useful in the following situations:

To disable automatically created firewall rules for LoadBalancer Services, you must specify the --disable-l4-lb-firewall-reconciliation flag when you create or update a cluster. The --disable-l4-lb-firewall-reconciliation flag does not affect other automatically created VPC firewall rules, such as those facilitating communication between nodes or those that permit health checks for your Services.

Limitations

GKE supports disabling the automatic creation of firewall rules for these types of LoadBalancer Services:

You cannot disable the automatic creation of firewall rules for these types of LoadBalancer Services:

  • Internal LoadBalancer Services not using GKE subsetting
  • Target pool-based external LoadBalancer Services

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Strategies for manual firewall rule management

Before you disable automatic creation of VPC firewall rules for LoadBalancer Services in your GKE cluster, work with a Security Admin to develop a strategy for configuring firewall rules manually.

  1. Decide which type of firewall policy to use: a hierarchical firewall policy, a global network firewall policy, or a regional network firewall policy. For steps to create a firewall policy, see:

    You can also use VPC firewall rules, which don't use any policy.

  2. Your manually created firewall rules must be ingress allow rules because the implied deny ingress firewall rule prohibits incoming traffic. When you've disabled the automatic creation of VPC firewall rules, incoming traffic won't reach your nodes unless you've created ingress allow firewall rules that match traffic for your LoadBalancer Services.

    Depending on the firewall rule's parameters, a single ingress allow firewall rule can apply to one or more LoadBalancer Services. For each ingress allow firewall rule you create, define the following parameters:

    • Target parameter: Ensure that the firewall rule at least includes all nodes of the cluster that contains the LoadBalancer Services. Supported targets depend on what type of firewall policy a rule is located in or if you're using a VPC firewall rule. For information about the target parameter of a rule in a firewall policy, see Targets.

    • Protocols and ports: Include all protocols and destination ports used by the LoadBalancer Services to which the firewall rule needs to apply.

    • Destination parameter: You can employ one of the following strategies for the destination parameter:

      • Include the IP addresses of all LoadBalancer Services to which the firewall rule needs to apply in the destination parameter. To find the IP address of a LoadBalancer Service, use the following command:
         kubectl get svc LOADBALANCER_NAME \
      -n NAMESPACE_NAME \
      -o jsonpath='{.status.loadBalancer.ingress[0].ip}
      • You can choose to omit the destination parameter. When the destination parameter is omitted, the target parameter defines the destinations implicitly. For more information, see Targets and IP addresses for ingress rules.

    • Source parameter: Specify the sources (for example, IP addresses) used by clients that need to connect to the Load Balancer Serfices to which the firewall rule needs to apply.

    For steps to create firewall rules, see:

  3. To ensure that your manually created firewall rules are working correctly, run a Network Intelligence Center (NIC) Connectivity Test. When running the Connectivity Test:

    • Set the destination to the IP address of the LoadBalancer Service.
    • Set the source to the IP address of the client.

    For more information, see Troubleshoot connectivity issues.

Disable creation of VPC firewall rules for your LoadBalancer Services

This section describes steps to disable the automatic creation of VPC firewall rules for LoadBalancer Services.

Create a new GKE cluster with VPC firewall rules creation disabled

  1. To disable the automatically created VPC firewall rules for LoadBalancer Services in a newly created cluster, create the cluster with the --disable-l4-lb-firewall-reconciliation flag:

    gcloud

    Autopilot:

    gcloud container clusters create-auto CLUSTER_NAME \
  --disable-l4-lb-firewall-reconciliation \
  --cluster-version=VERSION

    Standard:

    gcloud container clusters create CLUSTER_NAME \
  --disable-l4-lb-firewall-reconciliation \
  --enable-l4-ilb-subsetting \
  --cluster-version=VERSION

    Replace the following:

    • CLUSTER_NAME: the name of the new cluster.
    • VERSION: the GKE version.

    Terraform

    To create a cluster with VPC firewall rules creation disabled using Terraform, refer to the following example:

    resource "google_container_cluster" "primary" {
  provider = google-beta
  name = CLUSTER_NAME
  location = ZONE

  enable-l4-ilb-subsetting = true
  disable_l4_lb_firewall_reconciliation = true
}

  2. Create an external or internal LoadBalancer Service:

  3. Verify that GKE doesn't create an ingress allow firewall rule for the LoadBalancer Service. (Automatically created ingress allow firewall rules have names of the following form: k8s2-[cluster-id]-[namespace]-[service-name]-[suffixhash]).

    The following command returns nothing:

    gcloud compute firewall-rules list --format="value(name)" | grep "k8s2"

Update an existing GKE cluster to disable VPC firewall rules creation

Before disabling VPC firewall rules creation, note the following points about updating an existing cluster:

  • When you update an existing cluster to disable VPC firewall rules creation, GKE doesn't delete any existing firewall rules that GKE automatically created for your LoadBalancer Services.
  • GKE stops updating the existing rules and won't create new ones for the new LoadBalancer Services.
  • To turn VPC firewall rules creation back on, you can use the --enable-l4-lb-firewall-reconciliation flag with the gcloud_name container clusters update command.

To disable the automatic firewall rule creation on an existing cluster:

  1. Update the cluster to disable the automatic creation and management of firewall rules for LoadBalancer Services:

    gcloud container clusters update CLUSTER_NAME \
--disable-l4-lb-firewall-reconciliation \
--cluster-version=supported_version

    Replace the following:

    • CLUSTER_NAME: the name of the new cluster.
    • VERSION: the GKE version.

  2. Create an external or internal LoadBalancer Service:

  3. Verify that GKE doesn't create an ingress allow firewall rule for the LoadBalancer Service. (Automatically created ingress allow firewall rules have names of the following form: k8s2-[cluster-id]-[namespace]-[service-name]-[suffixhash]).

    The following command returns nothing:

    gcloud compute firewall-rules list --format="value(name)" | grep "k8s2"

Troubleshoot connectivity issues

The following examples illustrate how to use Network Intelligence Center Connectivity Tests to test connectivity to an external LoadBalancer Service:cluster:

  • Network Intelligence Center:

    1. In the Google Cloud console, go to the Network Intelligence Center and start a new connectivity test.
    2. From the drop-down menu, choose Any external public IP address as the source and select your load balancer from the destination.
    3. Re-run the connectivity test.

  • The gcloud CLI:

    The following example command creates and runs a test with your local workstation's public IP address as the source and the external load balancer's external IP address as the destination:

    gcloud network-management connectivity-tests create TEST_NAME \
--source-ip-address=SOURCE_IP_ADDRESS \
--source-network-type=NON_GCP_NETWORK \
--destination-ip-address=$(kubectl get svc LOADBALANCER_NAME -o jsonpath='{.status.loadBalancer.ingress[0].ip}') \
--destination-port=$(kubectl get svc LOADBALANCER_NAME -o jsonpath='{.spec.ports[0].targetPort}') \
--destination-network=projects/PROJECT_ID/global/networks/NETWORK_NAME

    Replace the following:

    • TEST_NAME: A name for the connectivity test.
    • SOURCE_IP_ADDRESS: The IP address of the system that needs to connect to the external LoadBalancer Service. For example
    • LOADBALANCER_NAME: The name of the external LoadBalancer Service.
    • PROJECT_ID: The project ID of the project that contains the cluster's VPC network. If your cluster uses a Shared VPC network, use the project ID of the host project.
    • NETWORK_NAME: The name of your cluster's VPC network.

    Check test results:

    gcloud network-management connectivity-tests describe TEST_NAME

What's next