Create a VPC-native cluster

This page explains how to configure VPC-native clusters in Google Kubernetes Engine (GKE).

To learn more about the benefits and requirements of VPC-native clusters, see the overview for VPC-native clusters.

For GKE Autopilot clusters, VPC-native networks are enabled by default and cannot be overridden.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Limitations

• You cannot convert a VPC-native cluster into a routes-based cluster, and you cannot convert a routes-based cluster into a VPC-native cluster.

• VPC-native clusters require VPC networks. Legacy networks are not supported.

• As with any GKE cluster, Service (ClusterIP) addresses are only available from within the cluster. If you need to access a Kubernetes Service from VM instances outside of the cluster, but within the cluster's VPC network and region, create an internal passthrough Network Load Balancer.
• If you use all of the Pod IP addresses in a subnet, you cannot replace the subnet's secondary IP address range without putting the cluster into an unstable state. However, you can create additional Pod IP address ranges using discontiguous multi-Pod CIDR.

Create a cluster

This section shows you how to complete the following tasks at cluster creation time:

• Create a cluster and subnet simultaneously.
• Create a cluster in an existing subnet.
• Create a cluster and select the control plane IP address range.
• Create a cluster with dual-stack networking in a new subnet (available in Autopilot clusters version 1.25 or later, and Standard clusters version 1.24 or later).
• Create a dual-stack cluster and a dual-stack subnet simultaneously (available in Autopilot clusters version 1.25 or later, and Standard clusters version 1.24 or later).

Create a cluster and subnet simultaneously

The following directions demonstrate how to create a VPC-native GKE cluster and subnet at the same time. The secondary range assignment method is managed by GKE when you perform these two steps with one command.

gcloud container clusters create CLUSTER_NAME \
    --location=COMPUTE_LOCATION \
    --enable-ip-alias \
    --create-subnetwork name=SUBNET_NAME,range=NODE_IP_RANGE \
    --cluster-ipv4-cidr=POD_IP_RANGE \
    --services-ipv4-cidr=SERVICES_IP_RANGE

{
  "name": CLUSTER_NAME,
  "description": DESCRIPTION,
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "createSubnetwork": true,
    "subnetworkName": SUBNET_NAME
  },
  ...
}

Create a cluster in an existing subnet

The following instructions demonstrate how to create a VPC-native GKE cluster in an existing subnet with your choice of secondary range assignment method.

module "gke" {
  source  = "terraform-google-modules/kubernetes-engine/google"
  version = "~> 12.0"

  project_id        = "PROJECT_ID"
  name              = "CLUSTER_NAME"
  region            = "COMPUTE_LOCATION"
  network           = "NETWORK_NAME"
  subnetwork        = "SUBNET_NAME"
  ip_range_pods     = "SECONDARY_RANGE_PODS"
  ip_range_services = "SECONDARY_RANGE_SERVICES"
}

{
  "name": CLUSTER_NAME,
  "description": DESCRIPTION,
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "clusterIpv4CidrBlock"      : string,
    "servicesIpv4CidrBlock"     : string,
    "clusterSecondaryRangeName" : string,
    "servicesSecondaryRangeName": string,

  },
  ...
}

Create a cluster and select the control plane IP address range

By default, clusters with Private Service Connect use the primary subnet range to provision the internal IP address assigned to the control plane endpoint. You can override this default setting by selecting a different subnet range during the cluster creation time only. The following sections show you how to create a cluster with Private Service Connect and override the subnet range.

gcloud container clusters create CLUSTER_NAME \
    --private-endpoint-subnetwork=SUBNET_NAME \
    --location=COMPUTE_LOCATION

gcloud container clusters create CLUSTER_NAME --enable-ip-alias \
    --enable-private-nodes  \
    --private-endpoint-subnetwork=SUBNET_NAME \
    --region=COMPUTE_REGION

Create a cluster with dual-stack networking

You can create a cluster with IPv4/IPv6 dual-stack networking on a new or existing dual-stack subnet. Dual-stack subnet is available in Autopilot clusters version 1.25 or later, and Standard clusters version 1.24 or later. Dual-stack subnet is not supported with Windows Server node pools.

Before setting up dual-stack clusters, we recommend that you complete the following actions:

• Learn more about the benefits and requirements of GKE clusters with dual-stack networking.
• See the restrictions and limitations of dual-stack networking.

In this section, you create a dual-stack subnet first and use this subnet to create a cluster.

1. To create a dual-stack subnet, run the following command:

   gcloud compute networks subnets create SUBNET_NAME \
       --stack-type=ipv4-ipv6 \
       --ipv6-access-type=ACCESS_TYPE \
       --network=NETWORK_NAME \
       --range=PRIMARY_RANGE \
       --region=COMPUTE_REGION
   

   Replace the following:

   • SUBNET_NAME: the name of the subnet that you choose.
   • ACCESS_TYPE: the routability to the public internet. Use INTERNAL for private clusters and EXTERNAL for public clusters. If --ipv6-access-type is not specified, the default access type is EXTERNAL.
   • NETWORK_NAME: the name of the network that will contain the new subnet. This network must meet the following conditions:
     • It must be a custom mode VPC network. For more information, see how to switch a VPC network from auto mode to custom mode.
     • If you replace the ACCESS_TYPE with INTERNAL, the network must use Unique Local IPv6 Unicast Addresses (ULA).
   • PRIMARY_RANGE: the primary IPv4 IP address range for the new subnet, in CIDR notation. For more information, see Subnet ranges.
   • COMPUTE_REGION: the compute region for the cluster.

2. To create a cluster with a dual-stack subnet, either use the gcloud CLI or the Google Cloud console:

Create a dual-stack cluster and a subnet simultaneously

You can create a subnet and a dual-stack cluster simultaneously. GKE creates an IPv6 subnet and assigns an external IPv6 primary range to the subnet.

If using Shared VPC, you cannot simultaneously create the cluster and subnet. Instead, a Network Admin in the Shared VPC host project must create the dual-stack subnet first.

• For Autopilot clusters, run the following command:

  gcloud container clusters create-auto CLUSTER_NAME \
      --location=COMPUTE_LOCATION \
      --network=NETWORK_NAME \
      --create-subnetwork name=SUBNET_NAME
  

  Replace the following:

  • CLUSTER_NAME: the name of your new Autopilot cluster.
  • COMPUTE_LOCATION: the Compute Engine location for the cluster.
  • NETWORK_NAME: the name of a VPC network that contains the subnet. This VPC network must be a custom mode VPC network that uses Unique Local IPv6 Unicast Addresses (ULA). For more information, see how to switch a VPC network from auto mode to custom mode.
  • SUBNET_NAME: the name of the new subnet. GKE can create the subnet based on your organization policies:
    • If your organization policies allow dual-stack, and the network is custom mode, GKE creates a dual-stack subnet and assigns an external IPv6 primary range to the subnet .
    • If your organization policies don't allow dual-stack, or if the network is in auto mode, GKE creates a single stack (IPv4) subnet.

• For Standard clusters, run the following command:

  gcloud container clusters create CLUSTER_NAME \
      --enable-ip-alias \
      --stack-type=ipv4-ipv6 \
      --ipv6-access-type=ACCESS_TYPE \
      --network=NETWORK_NAME \
      --create-subnetwork name=SUBNET_NAME,range=PRIMARY_RANGE \
      --location=COMPUTE_LOCATION
  

  Replace the following:

  • CLUSTER_NAME: the name of the new cluster that you choose.
  • ACCESS_TYPE: the routability to the public internet. Use INTERNAL for private clusters and EXTERNAL for public clusters. If --ipv6-access-type is not specified, the default access type is EXTERNAL.
  • NETWORK_NAME: the name of the network that will contain the new subnet. This network must meet the following conditions:
    • It must be a custom mode VPC network. For more information, see how to switch a VPC network from auto mode to custom mode.
    • If you replace the ACCESS_TYPE with INTERNAL, the network must use Unique Local IPv6 Unicast Addresses (ULA).
  • SUBNET_NAME: the name of the new subnet that you choose.
  • PRIMARY_RANGE: the primary IPv4 address range for the new subnet, in CIDR notation. For more information, see Subnet ranges.
  • COMPUTE_LOCATION: the Compute Engine location for the cluster.

Update the stack type

You can change the stack type of an existing cluster or a update an existing subnet to a dual-stack subnet.

Update the stack type on an existing cluster

Before you change the stack type on an existing cluster, consider the following limitations:

• Changing the stack type is supported in new GKE clusters running version 1.25 or later. GKE clusters that have been upgraded from versions 1.24 to versions 1.25 or 1.26 might get validation errors when enabling dual-stack network. In case of errors, contact the Google Cloud support team.

• Changing the stack type is a disruptive operation because GKE restarts components in both the control plane and nodes.

• GKE respects your configured maintenance windows when recreating nodes. This means that the cluster stack type won't be operational on the cluster until the next maintenance window occurs. If you prefer not to wait, you can manually upgrade the node pool by setting the --cluster-version flag to the same GKE version the control plane is already running. You must use the gcloud CLI if you use this workaround. For more information, see caveats for maintenance windows.

• Changing the stack type does not automatically change the IP family of existing Services. The following conditions apply:

  • If you change a single stack to dual-stack, the existing Services remain single stack.
  • If you change a dual-stack to single stack, the existing Services with IPv6 addresses get into an error state. Delete the Service and create one with the correct ipFamilies. To learn more, see an example of how to set up a Deployment.

To update an existing VPC-native cluster, you can use gcloud CLI or the Google Cloud console:

  gcloud container clusters update CLUSTER_NAME \
      --stack-type=STACK_TYPE \
      --location=COMPUTE_LOCATION

Update an existing subnet to a dual-stack subnet (available in Autopilot clusters version 1.25 or later, and Standard clusters version 1.24 or later).

Update an existing subnet to a dual-stack subnet

To update an existing subnet to a dual-stack subnet, run the following command. Updating a subnet does not affect any existing IPv4 clusters in the subnet.

gcloud compute networks subnets update SUBNET_NAME \
    --stack-type=ipv4-ipv6 \
    --ipv6-access-type=ACCESS_TYPE \
    --region=COMPUTE_REGION

Replace the following:

• SUBNET_NAME: the name of the subnet.
• ACCESS_TYPE: the routability to the public internet. Use INTERNAL for private clusters and EXTERNAL for public clusters. If --ipv6-access-type is not specified, the default access type is EXTERNAL.
• COMPUTE_REGION: the compute region for the cluster.

Verify the stack type, Pod, and Service IP address ranges

After you create a VPC-native cluster, you can verify its Pod and Service ranges.

gcloud container clusters describe CLUSTER_NAME

Advanced configuration for internal IP addresses

The following sections show how to use non-RFC 1918 private IP address ranges and how to enable privately used public IP address ranges.

Use non-RFC 1918 IP address ranges

GKE clusters can use IP address ranges outside of the RFC 1918 ranges for nodes, Pods, and Services. See valid ranges in the VPC network documentation for a list of non-RFC 1918 private ranges that can be used as internal IP addresses for subnet ranges.

This feature is not supported with Windows Server node pools.

Non-RFC 1918 IP address ranges are compatible with both private clusters and non-private clusters.

Non-RFC 1918 private ranges are subnet ranges — you can use them exclusively or in conjunction with RFC 1918 subnet ranges. Nodes, Pods, and Services continue to use subnet ranges as described in IP ranges for VPC-native clusters. If you use non-RFC 1918 ranges, keep the following in mind:

• Subnet ranges, even those using non-RFC 1918 ranges, must be assigned manually or by GKE before the cluster's nodes are created. You cannot switch to or cease using non-RFC 1918 subnet ranges unless you replace the cluster.

• Internal passthrough Network Load Balancers only use IP addresses from the subnet's primary IP address range. To create an internal passthrough Network Load Balancer with a non-RFC 1918 address, your subnet's primary IP address range must be non-RFC 1918.

Destinations outside your cluster might have difficulties receiving traffic from private, non-RFC 1918 ranges. For example, RFC 1112 (class E) private ranges are typically used as multicast addresses. If a destination outside of your cluster cannot process packets whose sources are private IP addresses outside of the RFC 1918 range, you can do the following:

• Use an RFC 1918 range for the subnet's primary IP address range. This way, nodes in the cluster use RFC 1918 addresses.

• Ensure that your cluster is running the IP masquerade agent and that the destinations are not in the nonMasqueradeCIDRs list. This way, packets sent from Pods have their sources changed (SNAT) to node addresses, which are RFC 1918.

Enable privately used external IP address ranges

GKE clusters can privately use certain external IP address ranges as internal, subnet IP address ranges. You can privately use any external IP address except for certain restricted ranges as described the VPC network documentation. This feature is not supported with Windows Server node pools.

Your cluster must be a VPC-native cluster in order to use privately used external IP address ranges. Routes-based clusters are not supported.

Privately used external ranges are subnet ranges. You can use them exclusively or in conjunction with other subnet ranges that use private addresses. Nodes, Pods, and Services continue to use subnet ranges as described in IP ranges for VPC-native clusters. Keep the following in mind when re-using external IP addresses privately:

• When you use a external IP address range as a subnet range, your cluster can no longer communicate with systems on the Internet that use that external range. The range becomes an internal IP address range in the cluster's VPC network.

• Subnet ranges, even those that privately use external IP address ranges, must be assigned manually or by GKE before the cluster's nodes are created. You cannot switch to or cease using privately used external IP addresses unless you replace the cluster.

GKE by default implements SNAT on the nodes to external IP destinations. If you have configured the Pod CIDR to use external IP addresses, the SNAT rules apply to Pod-to-Pod traffic. To avoid this you have 2 options:

• Create your cluster with the --disable-default-snat flag. For more details about this flag, refer to IP masquerading in GKE.
• Configure the configMap ip-masq-agent including in the nonMasqueradeCIDRs list at least the Pod CIDR, the Service CIDR, and the nodes subnet.

For Standard clusters, if the cluster version is 1.14 or later, both options will work. If your cluster version is earlier than 1.14, you can only use the second option (configuring ip-masq-agent).

What's next

• Read the GKE network overview.
• Learn about internal load balancing.
• Learn about configuring authorized networks.
• Learn about creating cluster network policies.