Disable the kubelet read-only port in GKE clusters

Autopilot Standard

This page shows you how to disable the insecure kubelet read-only port in Google Kubernetes Engine (GKE) clusters to reduce the risk of unauthorized access to the kubelet, and how to migrate applications to a more secure port.

In Kubernetes clusters, including GKE, the kubelet process running on nodes serves a read-only API using the insecure port 10255. Kubernetes doesn't perform any authentication or authorization checks on this port. The kubelet serves the same endpoints on the more secure, authenticated port 10250.

Disable the kubelet read-only port and switch any workloads that use port 10255 to use the more secure port 10250 instead.

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Requirements

You can only disable the insecure kubelet read-only port in GKE version 1.26.4-gke.500 or later.

Check for insecure port usage and migrate applications

Before you disable the insecure read-only port, migrate any of your running applications that use the port to the more secure read-only port. Workloads that might need migration include custom metrics pipelines and workloads that access kubelet endpoints.

For workloads that need access to the information served by the kubelet API on the node, like metrics, use port 10250.
For workloads that get Kubernetes information using the kubelet API on the node, like listing Pods on the node, use the Kubernetes API instead.

Check whether applications use the insecure kubelet read-only port

This section shows you how to check for insecure port usage in your cluster.

Check for port usage on Autopilot mode

To check for port usage in an Autopilot cluster, ensure that you have at least one workload that isn't a DaemonSet running in the cluster. If you perform the following steps on an empty Autopilot cluster, the results might be invalid.

Save the following manifest as read-only-port-metrics.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: node-metrics-printer-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-metrics-printer-role
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-metrics-printer-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: node-metrics-printer-role
subjects:
- kind: ServiceAccount
  name: node-metrics-printer-sa
  namespace: node-metrics-printer-namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-metrics-printer-sa
  namespace: node-metrics-printer-namespace
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-metrics-printer
  namespace: node-metrics-printer-namespace
spec:
  selector:
    matchLabels:
      app: node-metrics-printer
  template:
    metadata:
      labels:
        app: node-metrics-printer
    spec:
      serviceAccountName: node-metrics-printer-sa
      containers:
      - name: metrics-printer
        image: us-docker.pkg.dev/cloud-builders/ga/v1/curl:latest
        command: ["sh", "-c"]
        args:
        - 'while true; do curl -s --cacert "${CA_CERT}" -H "Authorization: Bearer $(cat ${TOKEN_FILE})" "https://${NODE_ADDRESS}:10250/metrics"|grep kubelet_http_requests_total; sleep 20; done'
        env:
        - name: CA_CERT
          value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        - name: TOKEN_FILE
          value: /var/run/secrets/kubernetes.io/serviceaccount/token
        - name: NODE_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP

This manifest does the following:

Creates a namespace and sets up RBAC roles to allow reading node metrics.
Deploys a DaemonSet that checks the kubelet metrics for the insecure read-only port.

Deploy the manifest:

kubectl create -f read-only-port-metrics.yaml

Check the DaemonSet logs:
```
kubectl logs --namespace=node-metrics-printer-namespace \
    --all-containers --prefix \
    --selector=app=node-metrics-printer
```
If the output has any results that contain the string server_type=readonly, an application is using the insecure read-only port.

Check for port usage on Standard mode

Run the following command on at least one node in every node pool in your cluster:

kubectl get --raw /api/v1/nodes/NODE_NAME/proxy/metrics | grep http_requests_total | grep readonly

Replace NODE_NAME with the name of the node.

If workloads on the node use the insecure kubelet read-only port, the output contains entries with the string server_type="readonly" like in the following example:

kubelet_http_requests_total{long_running="false",method="GET",path="healthz",server_type="readonly"} 3
kubelet_http_requests_total{long_running="false",method="GET",path="metrics",server_type="readonly"} 2549
kubelet_http_requests_total{long_running="false",method="GET",path="metrics/probes",server_type="readonly"} 2546
kubelet_http_requests_total{long_running="false",method="GET",path="other",server_type="readonly"} 2
kubelet_http_requests_total{long_running="false",method="GET",path="pods",server_type="readonly"} 1
kubelet_http_requests_total{long_running="false",method="GET",path="stats",server_type="readonly"} 2549

If the output is empty, no applications on that node use the insecure read-only port.

Migrate from the insecure kubelet read-only port

Typically, migrating an application to the secure port involves the following steps:

Update URLs or endpoints that refer to the insecure read-only port to use the secure read-only port instead. For example, change http://203.0.113.104:10255 to http://203.0.113.104:10250.
Set the certificate authority (CA) certificate of the HTTP client to the cluster CA certificate. To find this certificate, run the following command:
```
gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --format="value(masterAuth.clusterCaCertificate)"
```
Replace the following:
- CLUSTER_NAME: the name of your cluster.
- LOCATION: the location of your cluster.

The authenticated port 10250 requires that you grant appropriate RBAC roles to the subject to access the specific resources. For details, in the Kubernetes documentation, see kubelet authorization.

If your workload uses the /pods endpoint on the insecure kubelet read-only port, you need to grant the nodes/proxy RBAC permission to access the endpoint on the secure kubelet port. nodes/proxy is a powerful permission that you can't grant in GKE Autopilot clusters and that you shouldn't grant in GKE Standard clusters. Use the Kubernetes API with a fieldSelector for the node name instead.

If you use third-party applications that depend on the insecure kubelet read-only port, check with the application vendor for instructions to migrate to secure port 10250.

Example migration

Consider a Pod that queries metrics from the insecure kubelet read-only port.

apiVersion: v1
kind: Pod
metadata:
  name: kubelet-readonly-example
spec:
  restartPolicy: Never
  containers:
  - name: kubelet-readonly-example
    image: us-docker.pkg.dev/cloud-builders/ga/v1/curl:latest
    command:
      - curl
      - http://$(NODE_ADDRESS):10255/metrics
    env:
    - name: NODE_ADDRESS
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP

This application does the following:

Uses the default ServiceAccount in the default namespace
Runs the curl command against the /metrics endpoint on the node.

To update this Pod to use the secure port 10250, do the following steps:

Create a ClusterRole with access to get node metrics:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: curl-authenticated-role
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get

Bind the ClusterRole to your application's identity:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: curl-authenticated-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: curl-authenticated-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

Update the curl command to use the secure port endpoint with the corresponding authorization headers:

apiVersion: v1
kind: Pod
metadata:
  name: kubelet-authenticated-example
spec:
  restartPolicy: Never
  containers:
  - name: kubelet-readonly-example
    image: us-docker.pkg.dev/cloud-builders/ga/v1/curl:latest
    env:
    - name: NODE_ADDRESS
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    command:
    - sh
    - -c
    - 'curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization:
      Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://${NODE_ADDRESS}:10250/metrics'

Modify VPC firewall rules

If you update workloads to use port 10250, create firewall rules so that Pods in the cluster can reach the port in your node IP address ranges. The firewall rules should do the following:

Allow incoming traffic to TCP port 10250 on your node IP address ranges from internal Pod IP address ranges
Deny incoming traffic to TCP port 10250 on your node IP address ranges from the public internet.

You can use the following default GKE firewall rules as a template for the parameters to specify in your new rules.:

gke-[cluster-name]-[cluster-hash]-inkubelet
gke-[cluster-name]-[cluster-hash]-exkubelet

Disable the insecure read-only port on Autopilot clusters

You can disable the insecure kubelet read-only port for new and existing Autopilot clusters.

Disable the insecure read-only port on new Autopilot clusters

To disable the insecure kubelet read-only port when you create a new Autopilot cluster, use the --no-autoprovisioning-enable-insecure-kubelet-readonly-port flag, like in the following command:

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --no-autoprovisioning-enable-insecure-kubelet-readonly-port

Replace the following:

CLUSTER_NAME: the name of your new Autopilot cluster.
LOCATION: the location of your new Autopilot cluster.

Disable the insecure read-only port on existing Autopilot clusters

To disable the insecure kubelet read-only port on an existing Autopilot cluster, use the --no-autoprovisioning-enable-insecure-kubelet-readonly-port flag, like in the following command. All new and existing nodes in the cluster stop using the port.

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --no-autoprovisioning-enable-insecure-kubelet-readonly-port

Replace the following:

CLUSTER_NAME: the name of your existing cluster.
LOCATION: the location of your existing cluster.

Disable the insecure read-only port on Standard clusters

You can disable the insecure kubelet read-only port for entire Standard clusters or for individual node pools. We recommend that you disable the port for the entire cluster.

If you use node auto-provisioning, automatically provisioned node pools inherit the port setting that you specify at the cluster level. You can optionally specify a different setting for auto-provisioned node pools, but we recommend that you disable the port across all nodes in your cluster.

You can also use a node system configuration file to declaratively disable the insecure kubelet read-only port. If you use this file, you can't use the commands in the following sections to control the kubelet setting.

Disable the insecure read-only port on new Standard clusters

To disable the insecure kubelet read-only port on a new Standard cluster, use the --no-enable-insecure-kubelet-readonly-port flag like in the following command:

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --no-enable-insecure-kubelet-readonly-port

Replace the following:

CLUSTER_NAME: the name of your new Standard cluster.
LOCATION: the location of your new Standard cluster.

You can optionally add the --no-autoprovisioning-enable-insure-kubelet-readonly-port flag to separately control the node auto-provisioning setting, but we don't recommend this approach. This flag starts a rolling update of your auto-provisioned node pools, which might cause disruptions to your running workloads.

Disable the insecure read-only port on existing Standard clusters

To disable the insecure kubelet read-only port on an existing Standard cluster, use the --no-enable-insecure-kubelet-readonly-port flag like in the following command. Any new node pools won't use the insecure port. GKE doesn't update existing node pools automatically.

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --no-enable-insecure-kubelet-readonly-port

Replace the following:

CLUSTER_NAME: the name of your existing Standard cluster.
LOCATION: the location of your existing Standard cluster.

Disable the insecure read-only port on Standard node pools

We recommend that you set the read-only port setting at the cluster level in all cases. If you disabled the read-only port on an existing cluster that already had running node pools, use the following command to disable the port on those node pools.

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=LOCATION \
    --no-enable-insecure-kubelet-readonly-port

Replace the following:

NODE_POOL_NAME: the name of your node pool.
CLUSTER_NAME: the name of the cluster.
LOCATION: the location of the cluster.

Verify that the port is disabled

To verify that the insecure kubelet read-only port is disabled, describe the GKE resource.

Check the port status in Autopilot clusters

Run the following command:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=nodePoolAutoConfig \
    --format="value(nodeKubeletConfig)"

Replace the following:

CLUSTER_NAME: the name of your Autopilot cluster.
LOCATION: the location of your Autopilot cluster.

If the port is disabled, the output is the following:

insecureKubeletReadonlyPortEnabled: false

Check the port status in Standard clusters

The port status is available in the nodePoolDefaults.nodeConfigDefaults.nodeKubeletConfig field when you describe your cluster using the GKE API.

In Standard clusters, you'll also see a nodeConfig field that sets a value for the kubelet read-only port status. The nodeConfig field is deprecated and applies only to the default node pool that GKE creates when you create a new Standard mode cluster. The status of the port in the deprecated nodeConfig field doesn't apply to other node pools in the cluster.

Run the following command:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=nodePoolDefaults.nodeConfigDefaults \
    --format="value(nodeKubeletConfig)"

Replace the following:

CLUSTER_NAME: the name of your Standard cluster.
LOCATION: the location of your Standard cluster.

If the port is disabled, the output is the following:

insecureKubeletReadonlyPortEnabled: false

Check the port status in Standard node pools

Run the following command:

gcloud container node-pools describe NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=LOCATION \
    --flatten=config \
    --format="value(kubeletConfig)"

Replace the following:

NODE_POOL_NAME: the name of your node pool.
CLUSTER_NAME: the name of the cluster.
LOCATION: the location of the cluster.

If the port is disabled, the output is the following:

insecureKubeletReadonlyPortEnabled: false