Verify Google connections to the cluster control plane

This page describes how to verify connections made by Google personnel to your Google Kubernetes Engine (GKE) cluster control plane by correlating GKE logs with Access Transparency logs.

Access Transparency logs record the actions that Google personnel take when accessing your content. This guide is intended for security administrators who want additional verification of the contents of Access Transparency logs and associated Access Approval approvals by correlating with additional logging sources from GKE. This verification is entirely optional and is not required to secure your control plane.

Ensure that you're familiar with the following concepts:

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the control plane.

About Google access to cluster control plane instances

During troubleshooting sessions or for other justified business reasons, Google personnel like site reliability engineers and Cloud Customer Care employees might need administrative access to the Compute Engine instances that host your control plane. Depending on your Customer Care support package and configuration, Access Transparency provides detailed audit logging for this administrative access. Access Approval lets you require explicit approval before any Google personnel can access your resources. To learn more about administrative access and the tooling that you can use to authorize access and to record changes, see Administrative access for Google employees.

Control plane access logs

When you enable GKE control plane authority, GKE generates control plane access logs that you can optionally use to cross-reference the audit logs generated by Access Transparency and by Access Approval. GKE adds control plane access logs to the _Default bucket in Logging to record incoming network connections and specific SSH events in your control plane instances. You must enable GKE control plane authority in your project to generate control plane access logs for your clusters.

GKE generates the following access logs for the control plane:

The volume of control plane connection logs depends on factors like the number of nodes in the cluster, the number of control plane instances (regional clusters have more control plane instances than zonal clusters), and how often your workloads call the Kubernetes API server. The volume of SSH logs is small and depends on the number of node restarts.

To verify the connections to your control plane, you find the control plane access logs for your cluster and match those logs to audit logs from Access Transparency and Access Approval. This lets you confirm that all of the SSH connections to your control plane instances were as a result of authorized administrative access by Google personnel. When you enable GKE control plane authority for your cluster, all SSH access by Google personnel to your control plane is non-interactive, which means that every SSH connection runs a single command that you authorize.

Pricing

The following pricing considerations apply:

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Requirements

Control plane access logs require GKE version 1.31.1-gke.1846000 or later.

Required roles and permissions

To get the permissions that you need to enable log generation and to access and process logs, ask your administrator to grant you the following IAM roles:

  • Enable control plane connection logging in your cluster: Kubernetes Engine Cluster Admin (roles/container.clusterAdmin) on your project
  • Access logs and to use Logs Explorer and Log Analytics: Logs Viewer (roles/logging.viewer) on your project
  • Enable Access Transparency for the organization: Access Transparency Admin (roles/axt.admin) on your organization

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Enable GKE cluster control plane access logs

You can enable control plane access log generation for Autopilot mode and Standard mode clusters by enabling the corresponding logging component. For more information about control plane log types, see View GKE logs.

The supported logging component names for control plane access logs are as follows:

  • Control plane SSH logs: KCP_SSHD
  • Control plane connection logs: KCP_CONNECTION

Enable control plane access logs on a new cluster

The following example creates an Autopilot mode cluster with both types of control plane access logs enabled. To only enable one type of control plane access log, omit the corresponding component name from the command.

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --logging=SYSTEM,KCP_SSHD,KCP_CONNECTION

Replace the following:

  • CLUSTER_NAME: the name for the new cluster.
  • LOCATION: the location in which to create the cluster.

To specify logging components when you create a cluster using the GKE API, in the projects.locations.clusters.create method, set the corresponding values in the LoggingConfig object of the Cluster resource.

Enable control plane access logs on an existing cluster

To update the logging configuration of an existing cluster to enable control plane access logs, you must do the following:

  1. Find the existing logging components that your cluster uses.
  2. Identify the corresponding values to specify in the --logging flag in the gcloud CLI to keep those logging components enabled.
  3. Update your cluster logging configuration to enable the control plane access logs alongside your existing logging configuration.

The values that you specify for the --logging flag in the gcloud container clusters update command are different from the values that you see when you describe your cluster.

  1. Check the existing logging configuration of the cluster:

    gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

    The output is similar to the following:

    SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

  2. Identify the gcloud CLI values for the --logging flag that correspond to the logging component configuration from the output of the preceding step. For a list of the gcloud CLI values that correspond to specific logging components, see the Available logs table.

  3. Update the logging configuration with control plane access logs:

    gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --logging=SYSTEM,EXISTING_LOGS,KCP_ACCESS_LOGS

    Replace the following:

    • EXISTING_LOGS: a comma-separated list of logging components that your cluster already uses. Ensure that you specify the gcloud CLI values that correspond to these logging components, taken from the Available logs table.

    • KCP_ACCESS_LOGS: a comma-separated list of the control plane access log types to enable for the cluster, as follows:

      • For control plane SSH logs, specify KCP_SSHD.
      • For control plane connection logs, specify KCP_CONNECTION.

To specify logging components when you update a cluster using the GKE API, in the projects.locations.clusters.update method, set the existing and new logging component values in the LoggingConfig object of the ClusterUpdate resource.

Example cluster update to enable control plane access logs

Consider a cluster for which the gcloud container clusters describe command has the following logging configuration:

SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

The following cluster update command enables both types of control plane access logs while retaining the existing log configuration for this example cluster:

gcloud container clusters update example-cluster \
    --location=us-central1 \
    --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION

Cross-reference control plane access logs with Access Transparency logs

To verify control plane access for a cluster, get the control plane connection logs, the control plane SSH logs, and the Access Transparency logs for that cluster:

  1. In the Google Cloud console, open the Logs Explorer page.

    Go to Logs Explorer

  2. To get all of the logs for a specific cluster, including the control plane access logs and Access Transparency logs, run the following query:

    (logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection"
resource.labels.cluster_name="CLUSTER_NAME"
jsonPayload.connection.dest_port="22")
OR
(logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-sshd"
resource.labels.cluster_name="CLUSTER_NAME")
OR
(logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Faccess_transparency"
json_payload.accesses.methodName="GoogleInternal.SSH.Master"
json_payload.accesses.resourceName="//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME")

The output should display all of the following types of logs for your cluster:

  • Access Transparency log
  • Control plane connection log
  • SSH logs for each SSH session

Perform verification checks

Your primary verification check is whether you see all of the log types for any SSH connections when you run the Logging query from the preceding section. Every Access Transparency log should have a corresponding control plane connection log and one or more SSH logs. These logs are for actions that humans perform in your control plane instances, so the log volume should be small.

You can optionally perform the following additional checks of the log contents:

  1. For each control plane SSH log, check whether an Access Transparency log exists in a 15-minute window before the timestamp of the SSH log. This window of time accounts for the final SSH session closure happening several minutes after the initial connection was logged by Access Transparency.
  2. For each control plane connection log, check whether an Access Transparency log exists in a 5-minute window before the timestamp of the control plane connection log.

  3. If you use Access Approval for your cluster, check whether each Access Transparency log has a corresponding accessApprovals field. Cross-reference this field against Access Approval requests for your cluster.

    To get Access Approval requests for your project, see View historical Access Approval requests. Access Approval might be subject to exclusions.

  4. Optionally, validate the signature of the signed Access Approval associated with the Access Transparency log.

Control plane access log details

This section provides details and examples of the control plane access logs that GKE generates when Google personnel connect to your control plane instances.

Control plane connection logs

GKE adds a control plane connection log for every new incoming network connection to a control plane instance. These logs include specific details like the following:

  • Source and destination IP addresses and ports
  • Connection direction and protocol

The following example is of a control plane connection log:

{
  insertId: "z1eq8wonio335a5h",
  jsonPayload: {
    instance: {
      vm_name: "gke-dee49f0d6fa34ce3a2ac-f513-d195-vm",
      zone: "us-central1-c"
    },
    cluster: {
      cluster_id: "CLUSTER_ID",
      cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1-c/clusters/CLUSTER_NAME"
    },
    connection: {
      state: "NEW",
      src_ip: "192.0.2.100",
      src_port: 32774,
      dest_ip: "203.0.113.12",
      dest_port: 22,
      direction: "INGRESS"
      protocol: "TCP"
    },
  }
  logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection",
  receiveTimestamp: "2024-04-11T04:08:01.883070399Z",
  resource: {
    labels: {
      cluster_name: "CLUSTER_NAME",
      location: "us-central1-c",
      project_id: "PROJECT_ID"
    }
    type: "gke_cluster",
  }
  severity: "NOTICE",
  timestamp: "2024-04-11T04:07:59.019330Z"
}

The following fields in the log entry are relevant to verifying Google's actions:

  • cluster.cluster_urn: The fully-qualified resource identifier of the cluster. This identifier has the format //container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME, with the following variables:

    • PROJECT_NUMBER: the numerical project number of your clusters project.
    • LOCATION: the Google Cloud location of your cluster.
    • CLUSTER_NAME: the name of your cluster.

  • connection: Details about the connection attempt. This field has the following information:

    • state: the state of the connection. For new connections, the value is NEW.
    • src_ip: the IP address of the connection source.
    • src_port: the port number of the connection source.
    • dest_ip: the internal IP address of your control plane VM.
    • dest_port: the destination port number.
    • direction: the connection direction. The value is always INGRESS.
    • protocol: the IP protocol, like TCP.

Control plane SSH logs

GKE adds control plane SSH logs for events related to SSH connections to control plane instances. GKE records the following events:

  • SSH key accepted for a user
  • Session changed status from 0 to 1, indicating that the user successfully logged in
  • SSH session opened
  • SSH session closed
  • Session changed status from 1 to 0, indicating that the user logged out
  • SSH session failed

For example, the following control plane SSH log is for an SSH session being opened:

{
  insertId: "8llczemdulwbbwpa",
  jsonPayload: {
    instance: {
      vm_name: "gke-06cb920c609941c0a5ce-6840-40e9-vm",
      zone: "us-central1-c"
    },
    cluster: {
      cluster_id: "891e6d12889747748c1ac16ffcc6cb7c0a96450b36864eb680917c119fd801d0",
      cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1/clusters/CLUSTER_NAME",
    },
    message: "pam_unix(sshd:session): session opened for user REDACTED by (uid=0)",
  },
  logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-ssh",
  receiveTimestamp: "2024-04-09T13:21:55.231436462Z"
  resource: {
    type: "gke_cluster",
    labels: {
      cluster_name: "CLUSTER_NAME",
      location: "us-central1",
      project_id: "PROJECT_ID"
    }
  },
  severity: "NOTICE",
  timestamp: "2024-04-09T13:21:50.742246Z"
}

The following fields in the log entry are relevant to verifying Google's actions:

  • cluster.cluster_urn: The fully-qualified resource identifier of the cluster. This identifier has the format //container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME, with the following variables:

    • PROJECT_NUMBER: the numerical project number of your cluster project.
    • LOCATION: the Google Cloud location of your cluster.
    • CLUSTER_NAME: the name of your cluster.

  • message: Details about the SSH connection.

Disable control plane access logs

  1. To view the specific log types that your cluster uses, run the following command:

    gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

    The output is similar to the following:

    SYSTEM_COMPONENTS,WORKLOADS,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION

  2. To disable the control plane access logs for a cluster, run the following command:

    gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER

In the --logging flag, specify the logging components from the output of the previous command. This example command disables control plane access logs but keeps other control plane component logs enabled.

To disable logging components using the GKE API, set the corresponding values in the LoggingConfig object of the Cluster resource in the projects.locations.clusters method.

What's next