Audit logging

Overview

GKE on-prem makes use of Kubernetes Audit Logging, which keeps a chronological record of calls made to a cluster's Kubernetes API server. Audit logs are useful for investigating suspicious API requests and for collecting statistics.

Prior to GKE Enterprise 1.2, GKE on-prem wrote audit logs only to disk. GKE Enterprise 1.2 introduces an alpha feature that enables audit logs to be written to Cloud Audit Logs in a Google Cloud project. Writing to Cloud Audit Logs has several benefits over writing to disk, or even capturing logs in an on-premises logging system:

• Audit logs for all GKE clusters can be centralized.
• Log entries written to Cloud Audit Logs are immutable.
• Cloud Audit Logs entries are retained for 400 days.
• Cloud Audit Logs is included in the price of Anthos.

You can configure GKE on-prem to write logs to disk or to Cloud Audit Logs.

Disk-based audit logging

By default, audit logs in GKE on-prem are written to a persistent disk so that VM restarts and upgrades don't cause the logs to disappear. GKE on-prem retains up to 10 GB of audit log entries.

Cloud Audit Logs

If Cloud Audit Logs is enabled, then Admin Activity audit log entries from all Kubernetes API servers are sent to Google Cloud, using the project and location that you specify when you create a user cluster.

GKE on-prem deploys an audit-proxy Pod to the admin and user clusters. This Pod buffers and writes log entries to Cloud Audit Logs.

Limitations

Cloud Audit Logs for GKE on-prem is an alpha feature. This alpha release has several limitations:

• You can configure Cloud Audit Logs only during user cluster creation. That is, you must create a new user cluster.

• Data access logging is not supported.

• Modifying the Kubernetes audit policy is not supported.

These limitations will be addressed in future releases.

Enabling the Anthos GKE API

To use Cloud Audit Logs with GKE on-prem, you must enable the Anthos GKE API.

Enable the Anthos GKE API

Creating a service account for audit logging

You already have several service accounts that you created to use with GKE on-prem. For this alpha feature, you need to create an additional service account and have it allowlisted.

1. Create your Cloud Audit Logs service account:

   gcloud iam service-accounts create audit-logging-service-account

2. Create a JSON key file for your Cloud Audit Logs service account:

   gcloud iam service-accounts keys create audit-logging-key.json \
      --iam-account [Cloud Audit Logs_SERVICE_ACCOUNT_EMAIL]
   

   where [Cloud Audit Logs_SERVICE_ACCOUNT_EMAIL] is the email address of your Cloud Audit Logs service account.

3. Save audit-logging-key.json on the admin workstation in the same location as your other service account keys.

Allowlisting your service account

To request that your Cloud Audit Logs service account be allowlisted, fill in the Cloud Audit Logging for GKE On-Prem Alpha form. Your service account must be allowlisted before you create a user cluster that enables Cloud Audit Logs.

Creating a user cluster with Cloud Audit Logs enabled

1. If you have not yet created an admin cluster, create new admin and user clusters by following the instructions in Installing using static IP addresses or Installing using DHCP.

   If you already have an admin cluster, create a new user cluster by following the instructions in Creating additional user clusters.

2. After you run gkectl create-config, fill in your configuration file as usual, but also fill in the new cloudauditlogging section under usercluster.

3. Set cloudauditlogging.projectid to the project ID of the Google Cloud project where you want to view audit logs that pertain to your user cluster.

4. Set cloudauditloggijg.clusterlocation to a Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-prem data center.

5. Set cloudauditlogging.serviceaccountkeypath to the path of the JSON key file for your Cloud Audit Logs service account.

For example:

cloudauditlogging:
  projectid: "my-project"
  clusterlocation: "us-west1"
  serviceaccountkeypath: "/my-key-folder/audit-logging-key.json"

Continue the cluster creation as usual.

Accessing GKE on-prem audit logs

Disk-based audit logging

1. View the Kubernetes API servers running in your admin cluster and all of its associated user clusters:

   kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] get pods --all-namespaces -l component=kube-apiserver
   

   where [ADMIN_CLUSTER_KUBECONFIG] is the kubeconfig file of your admin cluster.

2. Download the API server's audit logs:

   kubectl cp -n [NAMESPACE] [APISERVER_POD_NAME]:/var/log/kube-audit/kube-apiserver-audit.log /tmp/kubeaudit.log
   

   This command fetches the latest log file, which can contain up to 1GB of data for the admin cluster and up to 850GB for user clusters.

   Older audit records are kept in separate files. To view those files:

   kubectl exec -n [NAMESPACE] [APISERVER_POD_NAME] -- ls /var/log/kube-audit -la
   

   Each audit log's filename has a timestamp that indicates when the file was rotated. A file contains audit logs up to that time and date.

Cloud Audit Logs

gcloud logging read \
    'logName="projects/[PROJECT_ID]/logs/externalaudit.googleapis.com%2Factivity" \
    AND resource.type="k8s_cluster" \
    AND protoPayload.serviceName="anthosgke.googleapis.com" ' \
    --limit 2 \
    --freshness 300d

Audit policy

Audit logging behavior is determined by a statically-configured Kubernetes audit logging policy. Changing this policy is currently not supported, but will be available in a future release.