This page explains how network isolation and access controls work for your Google Kubernetes Engine (GKE) cluster control plane and cluster nodes. This page replaces the page describing the concept of private clusters.
There are two aspects to consider when deciding how to configure network isolation:
- Control plane access which relates to who can access the cluster's control plane.
- Cluster networking which relates to isolating nodes.
This page shows you how to control and customize who can access the cluster's control plane and cluster nodes (in a Standard cluster) or workloads (in an Autopilot cluster). This customization is relevant when you are deciding the network configuration for your cluster. To go straight to defining your network configuration, see Customize your network isolation.
Plan and design your network isolation configuration with your organization's Network architects, Network engineers, Network administrators, or another team who is responsible for the definition, implementation, and maintenance of the network architecture.
Control plane access
In this section, you'll consider who can access your control plane.
Every GKE cluster has a control plane that handles Kubernetes API requests. The control plane runs on a virtual machine (VM) that is in a VPC network in a Google-managed project. A regional cluster has multiple replicas of the control plane, each of which runs on its own VM.
The control plane has two kinds of endpoints for cluster access:
- DNS-based endpoint
- IP-based endpoints
Use only the DNS-based endpoint to access your control plane for simplified configuration and a flexible and policy-based layer of security.
DNS-based endpoint
The DNS-based endpoint gives a unique DNS or fully qualified domain name (FQDN) for each cluster control plane. This DNS name can be used to access your control plane. The DNS name resolves to an endpoint that is accessible from any network reachable by Google Cloud APIs, including on-premises or other cloud networks. Enabling the DNS-based endpoint eliminates the need for a bastion host or proxy nodes to access the control plane from other VPC networks or external locations.
To access the control plane endpoint, you need to configure IAM roles and policies, and authentication tokens. For more details on the exact permissions required, see Customize your network isolation.
In addition to IAM policies and tokens, you can also configure IP or network-based controls with VPC Service Controls to secure access to the control plane. VPC Service Controls add another layer of access security with context-aware access based on attributes such as network origin. VPC Service Controls let you configure which networks can access the DNS endpoint. Using both IAM policies and VPC Service Controls to secure access to the DNS-based endpoint provides a multi-layer security model for your cluster control plane. VPC Service Controls are used by all Google Cloud APIs, aligning the security configuration of your clusters with your data hosted in all other Google Cloud APIs.
IP-based endpoints
Optionally, you can also configure access to the control plane using IP-based endpoints.
Terminology related to clusters and IP addresses
Google Cloud external IP addresses:
External IP addresses assigned to any VM used by any customer hosted on Google Cloud. Google Cloud owns these IP addresses. To learn more, see Where can I find Compute Engine IP ranges?
External IP addresses used by Google Cloud products such as Cloud Run or Cloud Run functions. Any client hosted on Google Cloud can instantiate these IP addresses. Google Cloud owns these IP addresses.
Google-reserved IP addresses: External IP addresses for GKE cluster management purposes. These IP addresses include GKE managed processes and other production Google services. Google owns these IP addresses.
GKE cluster IP address ranges: Internal IP addresses assigned to the cluster that GKE uses for the cluster's nodes, Pods, and Services.
Internal IP addresses: IP addresses from your cluster's VPC network. These IP addresses can include your cluster IP address, on-premises networks, the RFC 1918 ranges, or the privately used public IP (PUPI) addresses that include non-RFC 1918 ranges.
External endpoint: The external IP address that GKE assigns to the control plane.
Internal endpoint: The internal IP address that GKE assigns to the control plane.
VPC network: A virtual network in which you create subnets with IP address ranges specifically for the cluster's nodes and Pods.
When using IP-based endpoints, you have two options:
Expose the control plane on both the external and internal endpoints. This means that the control plane's external endpoint is accessible from an external IP address, and the internal endpoint is accessible from your cluster's VPC network. Nodes will communicate with the control plane on the internal endpoint only.
Expose the control plane on the internal endpoint only. This means that clients on the internet cannot connect to the cluster and the control plane is accessible from any IP address from your cluster's VPC network. Nodes will communicate with the control plane on the internal endpoint only.
This is the most secure option when using IP-based endpoints as it prevents all internet access to the control plane. This is a good choice if you have workloads that—for example—require controlled access due to data privacy and security regulations.
In both of the preceding options, you can restrict which IP addresses reach the endpoints by configuring authorized networks. If you use IP-based endpoints, then we strongly recommend that you add at least one authorized network. Authorized networks grant control plane access to a specific set of trusted IPv4 addresses, and provide protection and additional security benefits for your GKE cluster.
Enable authorized networks when using IP-based endpoints to secure your GKE cluster.
How authorized networks work
Authorized networks provide an IP-based firewall that controls access to the GKE control plane. Access to the control plane depends on the source IP addresses. When you enable authorized networks, you configure the IP addresses for which you want to allow access to the GKE cluster control plane endpoint as a CIDR block list.
The following table shows:
- The preset IP addresses that can always access the GKE control plane regardless of whether you enable authorized networks.
- The configurable IP addresses that can access the control plane when you allowlist them by enabling authorized networks.
| Control plane endpoints | Preset IP addresses that can always access the endpoints | Configurable IP address that can access the endpoints after enabling authorized networks |
|---|---|---|
| External and internal endpoints enabled |
|
|
| Only internal endpoint enabled |
|
|
With an authorized network, you can also configure the region from which an internal IP address can reach your control plane's internal endpoint. You can choose to allow access only from the cluster's VPC network, or from any Google Cloud region in a VPC or on-premises environment.
Limitations of using IP-based endpoints
- If you expand a subnet that is used by a cluster with authorized networks, you must update the authorized network configuration to include the expanded IP address range.
- If you have clients connecting from networks with dynamic IP addresses, such as employees on home networks, you must update the list of authorized networks frequently to maintain access to the cluster.
- Disabling access to the control plane's external endpoint prevents you from interacting with your cluster remotely. If you need to remotely access the cluster, you must use a bastion host which forwards client traffic to the cluster. In contrast, using a DNS-based endpoint only requires setting up IAM permissions.
- IP-based endpoints don't directly integrate with VPC Service Controls. VPC Service Controls operate at the service perimeter level to control data access and movement within Google Cloud. We recommend using both a DNS-based endpoint with VPC Service Controls for robust security defense.
- You can specify up to 100 authorized IP address ranges (external and internal IP addresses inclusive).
Cluster networking access
This section discusses isolating nodes within a Kubernetes cluster.
Enable private nodes
Prevent external clients from accessing nodes by provisioning those nodes only with internal IP addresses, making the nodes private. Workloads running on nodes without an external IP address cannot reach the internet unless NAT is enabled on the cluster's network. You can change these settings at any time.
You can enable private nodes at an individual cluster level or at the node pool (for Standard) or workload (for Autopilot) level. Enabling private nodes at the node pool or workload level overrides any node configuration at the cluster level.
If you update a public node pool to private mode, workloads that require access outside the cluster network might fail in the following scenarios:
Clusters in a Shared VPC network where Private Google Access is not enabled. Manually enable Private Google Access to ensure GKE downloads the assigned node image. For clusters that aren't in a Shared VPC network, GKE automatically enables Private Google Access.
Workloads that require access to the internet where Cloud NAT is not enabled or a custom NAT solution is not defined. To allow egress traffic to the internet, enable Cloud NAT or a custom NAT solution.
Whether or not you enable private nodes, the control plane still communicates with all nodes through internal IP addresses only.
Benefits of network isolation
The following are the benefits of network isolation:
Flexibility:
- Clusters have unified and flexible configuration. Clusters with or without external endpoints all share the same architecture and support the same functionality. You can secure access based on controls and best practices that meet your needs. All communication between the nodes in your cluster and the control plane use an internal IP address.
- You can change the control plane access and cluster node configuration settings at any time without having to re-create the cluster.
- You can choose to enable the external endpoint of the control plane if you need to manage your cluster from any location with internet access, or from networks or devices that aren't directly connected with your VPC. Or you can disable the external endpoint for enhanced security if you need to maintain network isolation for sensitive workloads. In either case, you can use authorized networks to limit access to trusted IP ranges.
Security:
- DNS-based endpoints with VPC Service Controls provide a multi-layer security model that protects your cluster against unauthorized networks as well as unauthorized identities accessing the control plane. VPC Service Controls integrate with Cloud Audit Logs to monitor access to the control plane.
- Private nodes, and the workloads running on these nodes, are not directly accessible from the public internet, significantly reducing the potential for external attacks on your cluster.
- You can block control plane access from Google Cloud external IP addresses or from external IP addresses to fully isolate the cluster control plane and reduce exposure to potential security threats.
Compliance: If you work in an industry with strict regulations for data access and storage, private nodes help with compliance by ensuring that sensitive data remains within your private network.
Control: Private nodes give you granular control over how traffic flows in and out of your cluster. You can configure firewall rules and network policies to allow only authorized communication. If you operate across multi-cloud environments, private nodes can help you establish secure and controlled communication between different environments.
Cost: By enabling private nodes, you can reduce costs for nodes that don't require an external IP address to access public services on the internet.
What's next
- Learn how to Customize your network isolation.
- Learn about Private Service Connect.