GKE Autopilot security capabilities


This page describes the security features, configurations, and settings in Google Kubernetes Engine (GKE) Autopilot. GKE Autopilot clusters implement many security configurations for you, which makes Autopilot the recommended way to run GKE.

This page is intended for Security specialists who want to understand the security restrictions that Google specifically applies to Autopilot clusters, and the security features that are available for use in Autopilot. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

Before reading this page, ensure that you're familiar with the following:

Security measures in Autopilot

Autopilot clusters enable and apply security best practices and settings by default, including many of the recommendations in the security overview and in Harden your cluster security.

If you want recommended resources based on your use case, skip to Security resources by use case. The following sections describe the security policies that Autopilot applies for you.

Autopilot and the Kubernetes Pod Security Standards

The Kubernetes project has a set of security guidelines named the Pod Security Standards that define the following policies:

  • Privileged: No access restrictions. Not used in Autopilot.
  • Baseline: Prevents known privilege escalation pathways. Allows most workloads to run without significant changes.
  • Restricted: Highest level of security. Requires significant changes to most workloads.

In Autopilot mode, GKE enforces Pod security constraints using a custom admission controller named GKE Warden. We enforce a default security policy that includes all of the recommendations in the Baseline level of the Pod Security Standards, with some modifications for usability. Additionally, the default GKE Warden admission policy for Autopilot enforces many constraints from the Restricted level of the Pod Security Standards, but avoids restrictions that would block a majority of your workloads from running.

If you need to apply additional restrictions to comply with the full Restricted policy, you can optionally use the PodSecurity admission controller in specific namespaces.

The following table describes how the controls in the default Autopilot GKE Warden policy compare to the controls in the Baseline and the Restricted levels of the Pod Security Standards. For descriptions of each control in this table, see the corresponding entry in Pod Security Standards.

When evaluating compliance, we considered how the constraints apply to your own workloads. This excludes verified Google Cloud partner workloads and system workloads that require specific privileges to function.

Control Baseline policy compliance Restricted policy compliance Additional information
HostProcess Autopilot blocks HostProcess.
Host namespaces Autopilot blocks host namespaces. Some containers from verified partners are allowed to use host namespaces.
Privileged containers Autopilot blocks privileged containers by default. Autopilot allows privileged containers from verified partners for purposes such as running security and monitoring tooling.
Linux capabilities

Autopilot workloads can only access the capabilities specified in the Baseline Pod Security Standard by default.

You can manually enable the following capabilities:

  • NET_RAW for ping and SYS_PTRACE for debugging: Add to Pod SecurityContext
  • NET_ADMIN for service meshes such as Istio: Specify --workload-policies=allow-net-admin in your cluster creation command. Available on new and upgraded existing clusters running GKE version 1.27 and later.

Autopilot also allows some verified partner workloads to set dropped capabilities.

HostPath volumes Partially complies Partially complies Autopilot allows containers to request read-only access to /var/log for debugging, but denies all other read or write access.
HostPorts Setting specific host ports is disallowed, which mitigates some scheduling issues, and prevents accidental or deliberate direct network exposure of services. You can manually set up random host port assignment from a known range to support direct-connection networking applications such as game servers.
AppArmor The AppArmor docker-default security profile is automatically applied to Container-Optimized OS.
SELinux SELinux isn't applied because AppArmor is already applied. SELinux is also not mandatory in the Pod Security Standards.
/proc mount type GKE does not set the ProcMountType feature flag. If the Pod securityContext sets procMount to "Unmasked", GKE automatically overrides it with "Default".
seccomp profile Autopilot applies the RuntimeDefault seccomp profile to all workloads. You can manually override this setting for specific workloads by setting the profile to Unconfined in the Pod specification.
sysctls GKE does not set the --allowed-unsafe-sysctls kubelet flag so pods with unsafe sysctls fail to schedule. For additional protection, as of 11 July 2023, new 1.27+ clusters also have a policy rule to enforce the securityContext settings and reject Pods that use unsafe sysctls.
Volume types Autopilot allows only the volume types in the Restricted policy with the following additions: HostPath volumes with read-only access to /var/log for debugging, gcePersistentDisk for Compute Engine persistent disks, and nfs for network file system volumes.
Privilege escalation This setting only provides protection to containers that aren't running as root. Industry surveys show that 76% of containers run as root, so Autopilot allows running as root to enable most workloads. This setting is also currently useful in de-privileging workloads to non-root by allowing use of file system capabilities to work around deficiencies with Kubernetes root capability handling.
Run as non-root Industry surveys show that 76% of containers run as root, so Autopilot allows running as root to enable most workloads.
Run as non-root user Containers can set runAsUser to 0. Industry surveys show that 76% of containers run as root, so Autopilot allows running as root to enable most workloads

Built-in security configurations

Google applies many built-in security settings to Autopilot clusters based on industry best practices and our expertise. The following table describes some of the security configurations that Autopilot applies for you:

Configuration Description
Host options
Linux capabilities

You can use the following Linux capabilities:

"SETPCAP", "MKNOD", "AUDIT_WRITE", "CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "SETGID", "SETUID", "NET_BIND_SERVICE", "SYS_CHROOT", "SETFCAP", "SYS_PTRACE"

You can also manually enable the following capabilities:

  • NET_RAW for ping: Add to Pod SecurityContext.
  • SYS_PTRACE for debugging: Add to Pod SecurityContext.
  • NET_ADMIN for service meshes such as Istio: Use --workload-policies=allow-net-admin when you create a cluster or update an existing cluster. After that, add the capability to the Pod SecurityContext. Available on GKE version 1.27 and later.

In GKE versions earlier than 1.21, the "SYS_PTRACE" capability is not supported.

Privileged containers Containers can't run in Privileged mode unless the container is deployed by a Google Cloud partner. Privileged containers can make changes to the underlying node, such as changing the kubelet. This access could increase the impact of a Pod compromise.
GKE-managed namespaces As a security measure, Autopilot doesn't allow deploying workloads in GKE-managed namespaces, such as kube-system.
Container isolation

Autopilot enforces the following restrictions on containers to limit the impact of container escape vulnerabilities.

Linux capabilities and kernel security

  • Autopilot applies the RuntimeDefault seccomp profile to all Pods in the cluster unless the Pods use GKE Sandbox. GKE Sandbox enforces host isolation and ignores seccomp rules specified in the Pod manifest. The sandbox is considered the security boundary for GKE Sandbox Pods.
  • Autopilot drops the CAP_NET_RAW Linux capability for all containers. This permission is not often used, and has been the subject of multiple escape vulnerabilities. The ping command might fail inside your containers because this capability is dropped. You can manually re-enable this capability by setting it in your Pod SecurityContext.
  • Autopilot drops the CAP_NET_ADMIN Linux capability for all containers. To re-enable this capability, specify the --workload-policies=allow-net-admin flag in your cluster creation or update command. NET_ADMIN is required by some workloads such as Istio.
  • Autopilot enables Workload Identity Federation for GKE, which prevents Pod access to sensitive metadata on the node.
  • Autopilot blocks Kubernetes Services that set the spec.externalIPs field to protect against CVE-2020-8554.
  • Autopilot allows only the following types of volumes:

    "configMap", "csi", "downwardAPI", "emptyDir", "gcePersistentDisk", "nfs", "persistentVolumeClaim", "projected", "secret"

    Other types of volumes are blocked because they require node privileges. HostPath volumes are blocked by default, but containers can request read-only access to /var/log paths for debugging.

Pod-level security policy enforcement Autopilot supports enforcement mechanisms for Pod-level security policies such as the PodSecurity admission controller, Gatekeeper, or Policy Controller. However, you might not need to use any of these if the built-in security configurations described on this page already meet your requirements.
SSH to nodes

Autopilot blocks SSH access to nodes. GKE handles all operational aspects of the nodes, including node health and all Kubernetes components running on the nodes.

You can still connect remotely to your running containers using the Kubernetes exec functionality to execute commands in your containers for debugging, including connecting to an interactive shell, for example with kubectl exec -it deploy/YOUR_DEPLOYMENT -- sh.

User impersonation GKE version 1.22.4-gke.1501 and later support user impersonation for all user-defined users and groups. System users and groups such as the kube-apiserver user and the system:masters group cannot be impersonated.
Mutating dynamic admission webhooks

Autopilot modifies mutating webhooks to exclude resources in managed namespaces, such as kube-system, from being intercepted.

Autopilot also rejects webhooks that specify one or more of the following resources, and any sub-resources of those resources.

- group: ""
  resource: nodes
- group: ""
  resource: persistentVolumes
- group: certificates.k8s.io
  resource: certificatesigningrequests
- group: authentication.k8s.io
  resource: tokenreviews

You can't use the * wildcard for resources or groups to bypass this restriction.

ValidatingAdmissionPolicies

Autopilot modifies ValidatingAdmissionPolicy objects to exclude resources in managed namespaces, such as kube-system, from being intercepted.

Autopilot also rejects ValidatingAdmissionPolicy objects that specify one or more of the following resources, and any sub-resources of those resources.

- group: ""
  resource: nodes
- group: ""
  resource: persistentVolumes
- group: certificates.k8s.io
  resource: certificatesigningrequests
- group: authentication.k8s.io
  resource: tokenreviews

You can't use the * wildcard for resources or groups to bypass this restriction.

Certificate signing requests You can create CertificateSigningRequests in Autopilot to create certificates that are signed by the cluster certificate authority. To prevent interference with system components, Autopilot rejects CertificateSigningRequests for known privileged identities, such as system groups, system agents, or Google-managed IAM service agents.
GKE security features Autopilot clusters enable recommended GKE security features for you. For a list of enabled and optional security features, refer to security features in Autopilot.
Node operating system Autopilot clusters use Container-Optimized OS with containerd as the node operating system. Container-Optimized OS is created and managed by Google.
GKE version upgrades Autopilot clusters are enrolled in a GKE release channel upon creation, and automatic upgrades are always enabled. Google automatically upgrades your control plane and nodes to the latest qualified version in the release channel over time.

Security boundaries in Autopilot

Autopilot provides access to the Kubernetes API but removes permissions to use some highly privileged Kubernetes features, such as privileged Pods. The goal is to limit the ability to access, modify, or directly control the node virtual machine (VM). Autopilot implements these restrictions to limit workloads from having low-level access to the node VM, so that Google Cloud can offer full management of nodes, and a Pod-level SLA.

Our intent is to prevent unintended access to the node VM. We accept submissions to that effect through the Google Vulnerability Reward Program (VRP) and will reward reports at the discretion of the Google VRP reward panel.

By design, privileged users such as cluster administrators have full control of any GKE cluster. As a security best practice, we recommend that you avoid granting powerful GKE or Kubernetes privileges widely and instead use namespace administrator delegation wherever possible as described in our multi-tenancy guidance.

Autopilot provisions single-tenant VMs in your project for your exclusive use. On each individual VM, your Autopilot workloads might run together, sharing a security-hardened kernel. Since the shared kernel represents a single security boundary, we recommend that if you require strong isolation, such as high-risk or untrusted workloads, run your workloads on GKE Sandbox Pods to provide multi-layer security protection.

Security resources based on use case

The following sections provide you with links and recommendations to plan, implement, and manage the security of your Autopilot clusters depending on your use case.

Plan cluster security

Use case Resources
Understand how GKE as a platform approaches security
Understand your role in hardening your environment Learn about the shared responsibility model.
View Google's recommendations for hardening measures and incident response
Understand how GKE implements audit logging

Authenticate and authorize

After setting up your Autopilot clusters, you might need to authenticate your users and applications to use resources such as the Kubernetes API or Google Cloud APIs.

Use case Resources
Authenticate users or applications to the cluster API server
  • To authenticate users, read Authenticating users.
  • To authenticate applications, read Authenticating applications, which provides steps for authenticating from apps in the same cluster, in other Google Cloud environments, or in external environments.
Authenticate applications to Google Cloud APIs and services Autopilot clusters let you use Workload Identity Federation for GKE to securely authenticate your workloads to Google Cloud APIs by configuring Kubernetes service accounts to act as IAM service accounts. For instructions, refer to Configure applications to use Workload Identity Federation for GKE.
Authorize actions at the project level To authorize actions across clusters at the project level, use IAM. You can grant or deny access to specific GKE and Kubernetes API resources using IAM roles and permissions. For instructions, refer to Create IAM policies.
Authorize actions at the cluster level To authorize actions on Kubernetes API resources in specific clusters, use the built-in Kubernetes role-based access control (RBAC) mechanism. For instructions, refer to Authorize actions in clusters using RBAC.
Authorize actions at the organization level You can use Google Cloud Organization Policy Service to enforce constraints on specific operations on GKE resources across your Google Cloud organization. For instructions, refer to Restrict actions on GKE resources using custom organization policies.

Harden clusters and workloads

If you have specialized isolation or hardening requirements beyond the pre-configured Autopilot measures, consider the following resources:

Use case Resources
Restrict public access to your cluster endpoint Configure the network isolation of your Autopilot clusters and disable the external endpoint of the cluster control plane. For instructions, refer to Configure the control plane access.
Restrict cluster access to specific networks Use control plane authorized networks to specify IP address ranges that can access your cluster.
Store sensitive information outside your cluster Storing sensitive data in an external, encrypted storage provider with versioning enabled is a common compliance requirement and a best practice. Use Secret Manager to store your data and access it from your Autopilot clusters using Workload Identity Federation for GKE. For instructions, refer to Access secrets stored outside GKE clusters using Workload Identity Federation for GKE.
Verify container images before deployment to your cluster Use Binary Authorization to check the integrity of the container images referenced in your Pod manifests at deploy time. For instructions, refer to Verify container images at deploy time using Binary Authorization.

Monitor your security posture

After setting up your clusters and deploying your workloads, you should set up and configure monitoring and logging so that you have observability over your cluster security posture. We recommend that you do all of the following:

  • Enroll your clusters in the GKE security posture dashboard to audit workloads for concerns such as problematic security configurations or vulnerabilities in your container operating system packages and get actionable mitigation information.
  • Get notified about new security bulletins and upgrade events using cluster notifications.
  • Observe your clusters using the GKE dashboard in Cloud Monitoring or the Observability tab in GKE.
  • Learn how to view and manage your GKE audit logs in Cloud Logging.

What's next