Audit logging

This page describes how to access Kubernetes apiserver audit logs.

Overview

Each GKE On-Prem cluster has Kubernetes Audit Logging, which keeps a chronological record of calls made to the cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests or for collecting statistics.

Disk-based audit logging

By default, audit logs from each API server are dumped to a persistent disk, so that VM restarts/upgrades won't cause the logs to disappear. GKE On-Prem retains up to 10GB of audit logs.

Cloud Audit logging

If Cloud Audit Logging is enabled, then Admin Activity audit logs from all API servers are sent to Google Cloud, using the project and location set during installation.

Accessing Kubernetes audit logs

Disk-based audit logging

You can only access audit logs through the admin cluster:

  1. View the Kubernetes API servers running in your clusters:

    kubectl get pods --all-namespaces -l component=kube-apiserver

  2. Download the API server's audit logs:

    kubectl cp -n [NAMESPACE] [APISERVER_POD_NAME]:/var/log/kube-audit/kube-apiserver-audit.log /tmp/kubeaudit.log

    This command fetches the latest log file, which can contain up to 1GB of data for the admin cluster and up to 850GB for user clusters.

    Older audit records are kept in separate files. To view those files:

    kubectl exec -n [NAMESPACE] [APISERVER_POD_NAME] -- ls /var/log/kube-audit -la

    Each audit log's filename has a timestamp that indicates when the file was rotated. A file contains audit logs up to that time and date.

Cloud Audit logging

Console

  1. In the Google Cloud console, go to the Logs page in the Logging menu.

    Go to the Logs page

  2. In the Filter by label or text search box, just above the drop-down menus discussed above, click the down arrow to open the drop-down menu. From the menu, choose Convert to advanced filter.

  3. Fill the text box with the following filter:

    resource.type="k8s_cluster"
logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.serviceName="anthosaudit.googleapis.com"

  4. Click Submit Filter to display all audit logs from GKE On-Prem clusters that where configured to log in to this project.

gcloud

List the first two log entries in your project's Admin Activity log that apply to the k8s_cluster resource type:

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity" \
    AND resource.type="k8s_cluster" \
    AND protoPayload.serviceName="anthosaudit.googleapis.com" ' \
    --limit 2 \
    --freshness 300d

where [PROJECT_ID] is your project ID.

The output shows two log entries. Notice that for each log entry, the logName field has the value projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Factivity and protoPayload.serviceName is equal to anthosaudit.googleapis.com.

Audit policy

Audit logging behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy is currently not supported.