Automatically audit workloads for configuration issues

This page shows you how to automatically audit your workload configurations for security concerns and get actionable recommendations to improve the security posture of your Google Kubernetes Engine (GKE) Autopilot and Standard clusters. This guides you through how to enable workload configuration auditing, deploy a test workload, view and action configuration audit results, and disable workload configuration auditing. It also covers the pricing, requirements, and limitations of workload configuration auditing.

This page is for Security specialists who monitor the security of their GKE clusters and want to learn more about how to automatically audit workload configurations in GKE Autopilot and Standard clusters. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

Before reading this page, because workload configuration auditing is a feature of the security posture dashboard, ensure that you're familiar with the following concepts:

• General overview of the security posture dashboard
• About workload configuration auditing

Pricing

The security posture dashboard is offered at no extra charge in GKE through the Container Security API.

Entries added to Cloud Logging use Cloud Logging pricing.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

• Enable the Container Security API.

  Enable Container Security API

Requirements

• To get the permissions that you need to use workload configuration auditing, ask your administrator to grant you the Security Posture Viewer (roles/containersecurity.viewer) IAM role on your Google Cloud project. For more information about granting roles, see Manage access to projects, folders, and organizations.

  This predefined role contains the permissions required to use workload configuration auditing. To see the exact permissions that are required, expand the Required permissions section:

  Required permissions

  The following permissions are required to use workload configuration auditing:

  • resourcemanager.projects.get
  • resourcemanager.projects.list
  • containersecurity.locations.list
  • containersecurity.locations.get
  • containersecurity.clusterSummaries.list
  • containersecurity.findings.list

  You might also be able to get these permissions with custom roles or other predefined roles.

• Workload configuration auditing requires GKE version 1.21 and later.

Enable workload configuration auditing

Workload configuration auditing is enabled by default in new Autopilot and Standard clusters running version 1.27 and later. You can also manually enable this feature using the gcloud CLI or the Google Cloud console.

Enable configuration auditing on a new cluster

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --security-posture=standard

Enable configuration auditing on an existing cluster

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --security-posture=standard

If you use Google Kubernetes Engine (GKE) Enterprise edition to manage fleets of clusters, you can also configure fleet-level configuration auditing settings that apply to all member clusters. For instructions, see Configure GKE security posture dashboard features at fleet-level.

Deploy a test workload

Deploy a sample application that intentionally violates the Pod Security Standards.

1. Save the following manifest as misconfig-sample.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: helloweb
     labels:
       app: hello
   spec:
     selector:
       matchLabels:
         app: hello
         tier: web
     template:
       metadata:
         labels:
           app: hello
           tier: web
       spec:
         containers:
         - name: hello-app
           image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
           ports:
           - containerPort: 8080
           securityContext:
             runAsNonRoot: false
           resources:
             requests:
               cpu: 200m
   

2. Deploy the application to your cluster:

   kubectl apply -f misconfig-sample.yaml
   

If you want to try other violations, modify misconfig-sample.yaml with the corresponding "bad" configuration.

View and action configuration audit results

The initial audit takes up to 15 minutes to return results. GKE displays the results on the security posture dashboard and automatically adds entries to the cluster logs.

View results

To see an overview of discovered concerns across your project's clusters and workloads, do the following:

1. Go to the Security Posture page in the Google Cloud console.

   Go to Security Posture

2. Click the Concerns tab.

3. In the Filter concerns pane, in the Concern type section, select the Configuration checkbox.

View concern details and recommendations

To view detailed information about a specific configuration concern, click the row containing that concern.

The Configuration Concern pane shows the following information:

• Description: a description of the concern.
• Recommended action: an overview of actions that you can take to fix the configuration issue. This section includes the following details:
  • Which resources need the fix
  • Sample commands that you can run to apply the fix to affected resources
  • The Google Cloud console instructions, if applicable, to fix the issue

View logs for discovered concerns

GKE adds entries to the _Default log bucket in Logging for each discovered concern. These logs are only retained for a specific period. For details, see Logs retention periods.

1. In the Google Cloud console, go to the Logs Explorer:

   Go to Logs Explorer

2. In the Query field, specify the following query:

   resource.type="k8s_cluster"
   jsonPayload.@type="type.googleapis.com/cloud.kubernetes.security.containersecurity_logging.Finding"
   jsonPayload.type="FINDING_TYPE_MISCONFIG"

3. Click Run query.

To receive notifications when GKE adds new findings to Logging, set up log-based alerts for this query. For more information, see Configure log-based alerts.

Clean up

1. Delete the sample workload that you deployed.

   kubectl delete deployment helloweb
   

2. Optionally, delete the cluster that you used.

   gcloud container clusters delete CLUSTER_NAME \
       --region=COMPUTE_REGION
   

Disable workload configuration auditing

You can disable workload configuration auditing using either the gcloud CLI or the Google Cloud console.

gcloud container clusters update CLUSTER_NAME \
    --region=LOCATION \
    --security-posture=disabled

Limitations of workload configuration auditing

• Windows Server node pools aren't supported.
• Workload configuration auditing doesn't scan GKE-managed workloads, such as workloads in the kube-system namespace.
• Workload configuration auditing is only available for clusters with less than 1,000 nodes.

• The security posture dashboard supports up to 150,000 active workload configuration auditing findings for each cluster. When the number of findings for a cluster exceeds this maximum, the security posture dashboard stops showing configuration findings for that cluster.

  To resolve this issue, use the logs in Logging to identify configuration issues and deploy updated manifests. When the number of configuration findings is less than 150,000, the security posture dashboard starts displaying findings for the cluster.

What's next

• Learn more about the security posture dashboard.
• Learn more about how configuration auditing works.
• Learn how to secure your clusters based on Google's recommendations.