Using Cloud DNS for GKE

This page explains how to use Cloud DNS as a Kubernetes DNS provider for Google Kubernetes Engine (GKE).

Using Cloud DNS as a DNS provider does not enable clients outside of a cluster to resolve and reach Kubernetes Services directly. You still need to expose Services externally using a Load Balancer and register their cluster external IP addresses on your DNS infrastructure.

For more information about using kube-dns as a DNS provider, see Service discovery and DNS.

To learn how to use a custom version of kube-dns or a custom DNS provider, see Setting up a custom kube-dns Deployment.

How Cloud DNS for GKE works

Cloud DNS can be used as the DNS provider for GKE, providing Pod and Service DNS resolution with managed DNS that does not require a cluster-hosted DNS provider. DNS records for Pods and Services are automatically provisioned in Cloud DNS for cluster IP address, headless and external name Services.

Cloud DNS supports the full Kubernetes DNS specification and provides resolution for A, AAAA, SRV and PTR records for Services within a GKE cluster. PTR records are implemented using response policy rules.

Using Cloud DNS as the DNS provider for GKE offers many benefits over cluster-hosted DNS:

• Removes overhead of managing the cluster-hosted DNS server. Cloud DNS requires no scaling, monitoring, or managing of DNS instances because it is a fully managed service hosted in the highly scalable Google infrastructure.
• Local resolution of DNS queries on each GKE node. Similar to NodeLocal DNSCache, Cloud DNS caches DNS responses locally, providing low latency and high scalability DNS resolution.

• Integration with Google Cloud Observability for DNS monitoring and logging. For more information, see Enabling and disabling logging for private managed zones.

Architecture

When Cloud DNS is the DNS provider for GKE, a controller runs as a GKE-managed Pod. This Pod runs on the control plane nodes of your cluster and syncs the cluster DNS records into a managed private DNS zone.

The following diagram shows how the Cloud DNS control plane and data plane resolve cluster names:

In the diagram, the Service backend selects the running backend Pods. The clouddns-controller creates a DNS record for Service backend.

The Pod frontend sends a DNS request to resolve the IP address of the Service named backend to the Compute Engine local metadata server at 169.254.169.254. The metadata server runs locally on the node, sending cache misses to Cloud DNS.

The Cloud DNS data plane runs locally within each GKE node or Compute Engine virtual machine (VM) instance. Depending on the type of Kubernetes Service, Cloud DNS resolves the Service name to its virtual IP address, for Cluster IP Services, or the list of endpoint IP addresses, for Headless Services.

After the Pod frontend resolves the IP address, the Pod can send traffic to the Service backend and any Pods behind the Service.

DNS scopes

Cloud DNS has the following DNS scopes. A cluster cannot operate in multiple modes simultaneously.

• GKE cluster scope: DNS records are only resolvable within the cluster, which is the same behavior as kube-dns. Only nodes running in the GKE cluster can resolve Service names. By default, clusters have DNS names that end in *.cluster.local. These DNS names are only visible within the cluster and don't overlap or conflict with *.cluster.local DNS names for other GKE clusters in the same project. This is the default mode.

  • Cloud DNS additive VPC scope:

    The Cloud DNS additive VPC scope is an optional feature that extends the GKE cluster scope to make headless Services resolvable from other resources in the VPC, such as Compute Engine VMs or on-premises clients connected using Cloud VPN or Cloud Interconnect. Additive VPC scope is an additional mode which is enabled alongside cluster scope, that you can enable or disable in your cluster without impacting DNS uptime or capability (cluster scope).

• VPC scope: DNS records are resolvable within the entire VPC. Compute Engine VMs and on-premise clients can connect using Cloud Interconnect or Cloud VPN and directly resolve GKE Service names. You must set a unique custom domain for each cluster, which means that all Service and Pod DNS records are unique within the VPC. This mode reduces communication friction between GKE and non-GKE resources.

The following table lists the differences between DNS scopes:

Cloud DNS resources

When you use Cloud DNS as your DNS provider for your GKE cluster, the Cloud DNS controller creates resources in Cloud DNS for your project. The resources that GKE creates depends on the Cloud DNS scope.

The naming convention used for these Cloud DNS resources is the following:

In addition to the zones mentioned in the previous table, the Cloud DNS controller creates the following zones in your project, depending on your configuration:

To learn more about how to create custom stub domains or custom upstream name servers, see Adding custom resolvers for stub domains.

Managed zones and forwarding zones

To serve internal DNS traffic, the Cloud DNS controller creates a managed DNS zone in each Compute Engine zone of the region the cluster belongs to.

For example, if you deploy a cluster in the us-central1-c zone, the Cloud DNS controller creates a managed zone in us-central1-a, us-central1-b, us-central1-c, and us-central1-f.

For each DNS stubDomain, the Cloud DNS controller creates one forwarding zone.

The Cloud DNS processes each DNS upstream using one managed zone with the . DNS name.

Pricing

When Cloud DNS is the DNS provider for GKE Standard clusters, DNS queries from Pods inside the GKE cluster are billed according to Cloud DNS pricing.

Queries to a VPC scoped DNS zone managed by GKE are billed using the standard Cloud DNS pricing.

Requirements

The Cloud DNS API must be enabled in your project.

Cloud DNS for GKE has the following requirements for cluster scope:

• For Standard, GKE version 1.24.7-gke.800, 1.25.3-gke.700 or later.

• For Autopilot, GKE version 1.25.9-gke.400, 1.26.4-gke.500 or later.
• Google Cloud CLI version 411.0.0 or later.

Cloud DNS for GKE has the following requirements for additive VPC scope:

• GKE version 1.28.3-gke.1430000 or later.

• Google Cloud CLI version 503.0.0 or later.
• The GKE cluster must use Cloud DNS cluster scope as the default DNS provider.

Cloud DNS for GKE has the following requirements for VPC scope:

• For Standard, GKE version 1.19 or later.
• Google Cloud CLI version 364.0.0 or later.
• The Cloud DNS API must be enabled in your project.

Restrictions and limitations

The following limitations apply:

• VPC scope is not supported on Autopilot clusters, only cluster scope is supported. If you need to resolve headless service names running in GKE Autopilot clusters, you must use additive VPC scope.

• You can enable additive VPC scope GKE Autopilot clusters only at cluster creation. Enabling or disabling additive VPC scope in existing GKE Autopilot clusters is not supported.

• Cloud DNS is not compliant with an Impact Level 4 (IL4) compliance regime. Cloud DNS for GKE cannot be used in an Assured Workload with an IL4 compliance regime. You need to use kube-dns in such regulated environment. For GKE Autopilot clusters, the selection of kube-dns or Cloud DNS is automatically done based on your compliance regime.

• Manual changes to the managed private DNS zones are not supported and are overwritten by the Cloud DNS controller. Modifications to the DNS records in those zones don't persist when the controller restarts.

• After you enable Cloud DNS for GKE in a cluster, kube-dns continues to run in the cluster. You can disable kube-dns by scaling the kube-dns Deployment and autoscaler to zero.

• You cannot change the DNS scope in a cluster after you have set the scope with the --cluster-dns-scope flag. If you need to change the DNS scope, recreate the cluster with a different DNS scope.

• Custom stub domains and upstream DNS server configurations apply to the DNS configurations of Pods and nodes. Pods using host networking or processes that run directly on the host also use the stub domain and upstream nameserver configurations. This is only supported in Standard.
• Custom stub domains and upstream nameservers configured through the kube-dns Configmap are automatically applied to Cloud DNS for cluster scope DNS. VPC scope DNS ignores the kube-dns ConfigMap and you must apply these configurations directly on Cloud DNS. This is only supported in Standard.

• There is no migration path from kube-dns to VPC scope, the operation is disruptive. Recreate the cluster when switching from kube-dns to VPC scope or conversely.
• For VPC scope, the secondary IP address range for Services must not be shared with any other clusters in that subnetwork.
• For VPC scope, the response policy associated with a PTR record is attached to the VPC network. If there are any other response policies bound to the cluster network, PTR record resolution fails for Kubernetes service IP addresses.

• If you try to create a headless Service with a number of Pods exceeding the allowed quota, Cloud DNS does not create record sets or records for the Service.

Quotas

Cloud DNS uses quotas to limit number of resources that GKE can create for DNS entries. Quotas and limits for Cloud DNS might be different from the limitations of kube-dns for your project.

The following default quotas are applied to each managed zone in your project when using Cloud DNS for GKE:

Resource limits

The Kubernetes resources that you create per cluster contribute to Cloud DNS resource limits, as described in the following table:

To learn more about how DNS records are created for Kubernetes, see Kubernetes DNS-Based Service Discovery.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

• Enable the Cloud DNS API in your project:

  Enable Cloud DNS API

Enable cluster scope DNS

In cluster scope DNS, only nodes running in the GKE cluster can resolve service names, and service names don't conflict between clusters. This behavior is the same as kube-dns in GKE clusters, which means that you can migrate clusters from kube-dns to Cloud DNS cluster scope without downtime or changes to your applications.

The following diagram shows how Cloud DNS creates a private DNS zone for a GKE cluster. Only processes and Pods running on the nodes in the cluster can resolve the cluster's DNS records, because only the nodes are in the DNS scope.

Enable cluster scope DNS in a new cluster

GKE Autopilot cluster

New Autopilot clusters in version 1.25.9-gke.400, 1.26.4-gke.500 or later default to Cloud DNS cluster scope.

GKE Standard cluster

You can create a GKE Standard cluster with Cloud DNS cluster scope enabled using the gcloud CLI or the Google Cloud console:

gcloud container clusters create CLUSTER_NAME \
    --cluster-dns=clouddns \
    --cluster-dns-scope=cluster \
    --location=COMPUTE_LOCATION

Enable cluster scope DNS in an existing cluster

GKE Autopilot cluster

You cannot migrate an existing GKE Autopilot cluster from kube-dns to Cloud DNS cluster scope. To enable Cloud DNS cluster scope, recreate the Autopilot clusters in version 1.25.9-gke.400, 1.26.4-gke.500 or later.

GKE Standard cluster

You can migrate an existing GKE Standard cluster from kube-dns to Cloud DNS cluster scope using the gcloud CLI or the Google Cloud console in a GKE Standard cluster.

When you migrate an existing cluster, the nodes in the cluster don't use Cloud DNS as a DNS provider until you recreate the nodes.

After you enable Cloud DNS for a cluster, the settings only apply if you upgrade existing node pools or you add new node pools to the cluster. When you upgrade a node pool, the nodes are recreated.

You can also migrate clusters that have running applications without interrupting cluster communication by enabling Cloud DNS as a DNS provider in each node pool separately. A subset of the nodes are operational at all times because some node pools use kube-dns and some node pools use Cloud DNS.

In the following steps, you enable Cloud DNS for a cluster and then upgrade your node pools. Upgrading your node pools recreates the nodes. The nodes then use Cloud DNS for DNS resolution instead of kube-dns.

Enable Cloud DNS additive VPC scope

This section describes steps to enable or disable Cloud DNS additive VPC scope, as an add-on to Cloud DNS cluster scope.

Enable Cloud DNS additive VPC scope in a new cluster

You can enable VPC scope DNS in a new GKE cluster using the gcloud CLI or the Google Cloud console.

For Autopilot

gcloud container clusters create-auto CLUSTER_NAME \
    --additive-vpc-scope-dns-domain=UNIQUE_CLUSTER_DOMAIN

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• UNIQUE_CLUSTER_DOMAIN: the name of a domain. You must ensure this name is unique within the VPC because GKE does not confirm this value. You cannot change this value after it is set. You must not use a domain that ends in ".local", or you might experience DNS resolution failures.

For Standard

gcloud container clusters create CLUSTER_NAME \
    --cluster-dns=clouddns \
    --cluster-dns-scope=cluster \
    --additive-vpc-scope-dns-domain=UNIQUE_CLUSTER_DOMAIN

The --cluster-dns-scope=cluster flag is optional because cluster is the default value.

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• UNIQUE_CLUSTER_DOMAIN: the name of a domain. You must ensure this name is unique within the VPC because GKE does not confirm this value. You cannot change this value after it is set. You must not use a domain that ends in ".local", or you might experience DNS resolution failures.

Enable Cloud DNS additive VPC scope in an existing cluster

To enable Cloud DNS additive VPC scope in an existing cluster, you first enable Cloud DNS for a cluster and then upgrade your node pools. Upgrading your node pools recreates the nodes. The nodes then use Cloud DNS for DNS resolution instead of kube-dns.

To enable Cloud DNS additive VPC scope in an existing cluster:

gcloud container clusters update CLUSTER_NAME \
    --additive-vpc-scope-dns-domain=UNIQUE_CLUSTER_DOMAIN \
    --location=COMPUTE_LOCATION

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• UNIQUE_CLUSTER_DOMAIN: the name of a domain. You must ensure this name is unique within the VPC because GKE does not confirm this value. You cannot change this value after it is set. You must not use a domain that ends in ".local", or you might experience DNS resolution failures.
• COMPUTE_LOCATION: the Compute Engine location for the cluster.

Enable VPC scope DNS

In VPC scope DNS, a cluster's DNS names are resolvable within the entire VPC. Any client in the VPC can resolve cluster DNS records.

VPC scope DNS enables the following use cases:

• Headless service discovery for non-GKE clients within the same VPC.
• GKE Service resolution from on-premise or 3rd party cloud clients. For more information, see Inbound server policy.
• Service resolution where a client can decide which cluster they want to communicate with using the custom cluster DNS domain.

In the following diagram, two GKE clusters use VPC scope DNS in the same VPC. Both clusters have a custom DNS domain, .cluster1 and .cluster2, instead of the default .cluster.local domain. A VM communicates with the headless backend Service by resolving backend.default.svc.cluster1. Cloud DNS resolves the headless Service to the individual Pod IPs in the Service and the VM communicates directly with the Pod IP addresses.

You can also perform this type of resolution from other networks when connected to the VPC through Cloud Interconnect or Cloud VPN. DNS server policies enable clients from networks connected to the VPC to resolve names in Cloud DNS, which includes GKE Services if the cluster is using VPC scope DNS.

Enable VPC scope DNS in an existing cluster

The migration is supported in GKE Standard only and not on GKE Autopilot.

GKE Autopilot cluster

You cannot migrate a GKE Autopilot cluster from kube-dns to Cloud DNS VPC scope.

GKE Standard cluster

You can migrate an existing GKE cluster from kube-dns to Cloud DNS VPC scope using the gcloud CLI or the Google Cloud console.

After you enable Cloud DNS for a cluster, the settings only apply if you upgrade existing node pools or you add new node pools to the cluster. When you upgrade a node pool, the nodes are recreated.

In the following steps, you enable Cloud DNS for a cluster and then upgrade your node pools. Upgrading your node pools recreates the nodes. The nodes then use Cloud DNS for DNS resolution instead of kube-dns.

Verify Cloud DNS

Verify that Cloud DNS for GKE is working correctly for your cluster:

1. Verify that your nodes are using Cloud DNS by connecting to a Pod on a node and running the command cat /etc/resolv.conf:

   kubectl exec -it POD_NAME -- cat /etc/resolv.conf | grep nameserver
   

   Replace POD_NAME with the name of the Pod.

   Based on the cluster mode, the output is similar to the following:

   GKE Autopilot cluster

   nameserver 169.254.20.10
   

   Because the NodeLocal DNSCache is enabled by default in GKE Autopilot, the Pod is using NodeLocal DNSCache.

   Only when the local cache does not have an entry for the name being looked up, NodeLocal DNSCache forwards the request to Cloud DNS.

   GKE Standard cluster

   nameserver 169.254.169.254
   

   The Pod is using 169.254.169.254 as the nameserver, which is the IP address of the metadata server where the Cloud DNS data plane listens for requests on port 53. The nodes no longer use the kube-dns Service address for DNS resolution and all DNS resolution occurs on the local node.

   If the output is an IP address similar to 10.x.y.10, then the Pod is using kube-dns. See the Troubleshooting section to understand why your pod is still using kube-dns.

   If the output is 169.254.20.10, it means that you have enabled NodeLocal DNSCache in your cluster, then the Pod is using NodeLocal DNSCache.

2. Deploy a sample application to your cluster:

   kubectl run dns-test --image us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0
   

3. Expose the sample application with a Service:

   kubectl expose pod dns-test --name dns-test-svc --port 8080
   

4. Verify that the Service deployed successfully:

   kubectl get svc dns-test-svc
   

   The output is similar to the following:

   NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   dns-test-svc   ClusterIP   10.47.255.11    <none>        8080/TCP   6m10s
   

   The value of CLUSTER-IP is the virtual IP address for your cluster. In this example, the virtual IP address is 10.47.255.11.

5. Verify that your Service name was created as a record in the private DNS zone for your cluster:

   gcloud dns record-sets list \
       --zone=PRIVATE_DNS_ZONE \
       --name=dns-test-svc.default.svc.cluster.local.
   

   Replace PRIVATE_DNS_ZONE with the name of the managed DNS zone.

   The output is similar to the following:

   NAME: dns-test-svc.default.svc.cluster.local.
   TYPE: A
   TTL: 30
   DATA: 10.47.255.11
   

Disable Cloud DNS for GKE

GKE Autopilot cluster

You cannot disable Cloud DNS in a GKE Autopilot cluster that was created with Cloud DNS by default. See the requirements for more information about GKE Autopilot clusters using Cloud DNS by default.

GKE Standard cluster

You can disable Cloud DNS cluster scope using the gcloud CLI or the Google Cloud console in a GKE Standard cluster.

gcloud container clusters update CLUSTER_NAME \
    --cluster-dns=default

gcloud container clusters upgrade dns --node-pool=DEFAULT_POOL \
    --cluster-version=VERSION \
    --region=REGION

Disable Cloud DNS additive VPC scope

When you disable Cloud DNS additive VPC scope for your cluster, only the DNS records in the private zones attached to the VPC network will be deleted. The records in the private DNS zones for the GKE cluster will remain, managed by the Cloud DNS for GKE, until the headless service is deleted from the cluster.

To disable Cloud DNS additive VPC scope, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --disable-additive-vpc-scope

Replace CLUSTER_NAME with the name of the cluster.

This will keep your cluster with Cloud DNS cluster scope enabled to provide DNS resolution from within the cluster.

Clean up

After completing the exercises on this page, follow these steps to remove resources and prevent unwanted charges incurring on your account:

1. Delete the service:

   kubectl delete service dns-test-svc
   

2. Delete the Pod:

   kubectl delete Pod dns-test
   

3. You can also delete the cluster.

Use Cloud DNS with Shared VPC

Cloud DNS for GKE supports Shared VPC for both VPC and cluster scope.

The GKE controller creates a managed private zone in the same project as the GKE cluster.

The GKE service account for the GKE cluster does not require Identity and Access Management (IAM) for DNS outside of its own project because the managed zone and GKE cluster reside within the same project.

More than one cluster per service project

Starting in GKE versions 1.22.3-gke.700, 1.21.6-gke.1500, you can create clusters in multiple service projects referencing a VPC in the same host project.

If you already have clusters using Shared VPC and Cloud DNS VPC scope, you must manually migrate them with the following steps:

• Upgrade your existing clusters that have Shared VPC enabled to GKE version 1.22.3-gke.700+ or 1.21.6-gke.1500+.
• Migrate the response policy from the service project to the host project. You only need to perform this migration once per Shared VPC network.

You can migrate your response policy using the Google Cloud console.

Perform the following steps in your service project:

1. Go to the Cloud DNS zones page.

   Go to Cloud DNS zones

2. Click the Response policy zones tab.

3. Click the response policy for your VPC network. You can identify the response policy by the description, which is similar to "Response policy for GKE clusters on network NETWORK_NAME."

4. Click the In use by tab.

5. Next to the name of your host project, click delete to remove the network binding.

6. Click the Response policy rules tab.

7. Select all of the entries in the table.

8. Click Remove response policy rules.

9. Click delete Delete response policy.

After you delete the response policy, the DNS controller creates the response policy in the host project automatically. Clusters from other service projects share this response policy.

Support custom stub domains and upstream name servers

Cloud DNS for GKE supports custom stub domains and upstream name servers configured using kube-dns ConfigMap. This support is only available for GKE Standard clusters.

Cloud DNS translates stubDomains and upstreamNameservers values into Cloud DNS forwarding zones.

Known issues

Terraform plans to recreate Autopilot cluster due to dns_config change

If you use terraform-provider-google or terraform-provider-google-beta, you might experience an issue where Terraform tries to recreate an Autopilot cluster. This error occurs because newly created Autopilot clusters running version 1.25.9-gke.400, 1.26.4-gke.500, 1.27.1-gke.400 or later use Cloud DNS as a DNS provider instead of kube-dns.

This issue is resolved in version 4.80.0 of the Terraform provider of Google Cloud.

If you cannot update the version of terraform-provider-google or terraform-provider-google-beta, you can add lifecycle.ignore_changes to the resource to ensure that google_container_cluster ignores changes to dns_config:

  lifecycle {
    ignore_changes = [
      dns_config,
    ]
  }

DNS resolution failing after migrating from kube-dns to Cloud DNS with NodeLocal DNSCache enabled

This section describes a known issue present in GKE clusters in the Cloud DNS with NodeLocal DNSCache in Cluster Scope.

After you migrate from kube-DNS to Cloud DNS with NodeLocal DNSCache enabled on the cluster, your cluster might experience intermittent resolution errors.

While using kube-dns with NodeLocal DNSCache being enabled on the cluster, NodeLocal DNSCache is configured to listen on both addresses (NodeLocal DNSCache address and kube-dns address).

To check the status of NodeLocal DNSCache, run the following command:

kubectl get cm -n kube-system node-local-dns -o json | jq .data.Corefile -r | grep bind

The output is similar to the following:

    bind 169.254.20.10 x.x.x.10
    bind 169.254.20.10 x.x.x.10

After updating the cluster to Cloud DNS the NodeLocal DNSCache configuration is changed, check the NodeLocal DNSCache:

kubectl get cm -n kube-system node-local-dns -o json | jq .data.Corefile -r | grep bind

The output is similar to the following:

    bind 169.254.20.10
    bind 169.254.20.10

The following workflow explains the entries in resolv.conf file while migration and node recreation:

Before migration

• Pods have resolv.conf configured to kube-dns-IP (i.e. x.x.x.10).
• Node-local-cache Pods listen on both addresses (bind 169.254.20.10 x.x.x.10) and intersects DNS requests from Pods.
• NodeLocal DNSCache works as a cache and little stress is put on kube-dns pods.

After migration

• After the control plane updates to use Cloud DNS, the Pods still have resolv.conf configured to kube-dns-IP (i.e. x.x.x.10). This configuration remains because GKE requires node recreation to use 169.254.20.10 .Cloud DNS setup requires 169.254.20.10
• Node-local-cache Pods listen on the NodeLocal DNSCache address only (bind 169.254.20.10). Request doesn't go to Node-local-cache Pods.
• All requests from Pods are directly sent to kube-dns Pods. This setup generates high traffic on the Pods.

After nodes recreation or node pool upgrade

• Pods have resolv.conf configured to NodeLocal DNSCache IP address (169.254.20.10).
• Node-local-cache Pods listen on the NodeLocal DNSCache address only (bind 169.254.20.10) and receive DNS requests from Pods on this IP address.

When node pools use kube-dns IP under resolv.conf before the node pool recreation, an increase in the DNS query traffic also increases traffic on the kube-dns pods, which causes intermittent failure of DNS requests. To minimize errors, you must plan this migration during downtime periods.

Troubleshooting

For information about troubleshooting Cloud DNS, see the following pages:

• For advice about Cloud DNS in GKE, see Troubleshoot Cloud DNS in GKE.
• For advice specifically about Cloud DNS, see Troubleshoot Cloud DNS.
• For general advice about diagnosing Kubernetes DNS issues, see Debugging DNS Resolution.

What's next

• Read an overview of how GKE provides managed DNS.
• Read DNS for services and Pods for a general overview of how DNS is used in Kubernetes clusters.
• Learn about Scopes and hierarchies in Cloud DNS.
• Learn about Enabling and disabling logging for private managed zones.